nornor02 commited on
Commit
54f1ebe
·
verified ·
1 Parent(s): 32fac50

Training in progress, step 4000

Browse files
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:91f02f501f1865950c5b71eac1056d9f4e69a7df89f1a18e95f167d7784c95e7
3
  size 328693404
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc81d7660fc71a769834227c71601030cdd1445903a5c8bbadf85ae1146e71bf
3
  size 328693404
runs/Jan12_02-28-34_ea19ce76d6cc/events.out.tfevents.1705026515.ea19ce76d6cc.10822.0 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c2664f8416b6b127799ad8e98c3a05c524a983e995d4ee8b74443adcd36ba85e
3
- size 5464
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3f8a50bb8a70a34e43d10755db4a358b5f20c661f3b1b27a8180d547734f937e
3
+ size 5621
wandb/debug-internal.log CHANGED
@@ -368,3 +368,45 @@
368
  2024-01-12 02:39:01,870 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
369
  2024-01-12 02:39:03,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
370
  2024-01-12 02:39:06,871 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
368
  2024-01-12 02:39:01,870 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
369
  2024-01-12 02:39:03,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
370
  2024-01-12 02:39:06,871 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
371
+ 2024-01-12 02:39:08,304 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
372
+ 2024-01-12 02:39:09,305 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
373
+ 2024-01-12 02:39:11,877 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
374
+ 2024-01-12 02:39:14,306 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
375
+ 2024-01-12 02:39:16,880 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
376
+ 2024-01-12 02:39:19,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
377
+ 2024-01-12 02:39:21,881 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
378
+ 2024-01-12 02:39:24,308 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
379
+ 2024-01-12 02:39:26,882 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
380
+ 2024-01-12 02:39:29,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
381
+ 2024-01-12 02:39:31,884 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
382
+ 2024-01-12 02:39:34,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
383
+ 2024-01-12 02:39:36,885 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
384
+ 2024-01-12 02:39:38,305 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
385
+ 2024-01-12 02:39:40,306 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
386
+ 2024-01-12 02:39:41,890 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
387
+ 2024-01-12 02:39:45,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
388
+ 2024-01-12 02:39:46,892 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
389
+ 2024-01-12 02:39:50,308 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
390
+ 2024-01-12 02:39:51,893 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
391
+ 2024-01-12 02:39:55,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
392
+ 2024-01-12 02:39:56,894 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
393
+ 2024-01-12 02:40:00,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
394
+ 2024-01-12 02:40:01,895 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
395
+ 2024-01-12 02:40:05,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
396
+ 2024-01-12 02:40:06,896 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
397
+ 2024-01-12 02:40:08,306 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
398
+ 2024-01-12 02:40:11,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
399
+ 2024-01-12 02:40:11,898 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
400
+ 2024-01-12 02:40:14,829 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: partial_history
401
+ 2024-01-12 02:40:14,831 DEBUG SenderThread:10874 [sender.py:send():382] send: history
402
+ 2024-01-12 02:40:14,831 DEBUG SenderThread:10874 [sender.py:send_request():409] send_request: summary_record
403
+ 2024-01-12 02:40:14,832 INFO SenderThread:10874 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
404
+ 2024-01-12 02:40:15,413 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json
405
+ 2024-01-12 02:40:16,413 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
406
+ 2024-01-12 02:40:16,832 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
407
+ 2024-01-12 02:40:17,119 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
408
+ 2024-01-12 02:40:18,414 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
409
+ 2024-01-12 02:40:22,122 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
410
+ 2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
+ 2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
+ 2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240112_022837-twhkj0lf/files/output.log CHANGED
@@ -20,4 +20,7 @@ Checkpoint destination directory /kaggle/working/checkpoint-3000 already exists
20
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
21
  warnings.warn('Was asked to gather along dimension 0, but all '
22
  Checkpoint destination directory /kaggle/working/checkpoint-3500 already exists and is non-empty.Saving will proceed but saved results may be invalid.
 
 
 
23
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
 
20
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
21
  warnings.warn('Was asked to gather along dimension 0, but all '
22
  Checkpoint destination directory /kaggle/working/checkpoint-3500 already exists and is non-empty.Saving will proceed but saved results may be invalid.
23
+ /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
24
+ warnings.warn('Was asked to gather along dimension 0, but all '
25
+ Checkpoint destination directory /kaggle/working/checkpoint-4000 already exists and is non-empty.Saving will proceed but saved results may be invalid.
26
  /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 0.6047, "train/learning_rate": 6.538461538461539e-06, "train/epoch": 269.23, "train/global_step": 3500, "_timestamp": 1705027131.5049787, "_runtime": 613.8460266590118, "_step": 6}
 
1
+ {"train/loss": 0.5064, "train/learning_rate": 4.615384615384616e-06, "train/epoch": 307.69, "train/global_step": 4000, "_timestamp": 1705027214.829182, "_runtime": 697.1702299118042, "_step": 7}
wandb/run-20240112_022837-twhkj0lf/logs/debug-internal.log CHANGED
@@ -368,3 +368,45 @@
368
  2024-01-12 02:39:01,870 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
369
  2024-01-12 02:39:03,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
370
  2024-01-12 02:39:06,871 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
368
  2024-01-12 02:39:01,870 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
369
  2024-01-12 02:39:03,311 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
370
  2024-01-12 02:39:06,871 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
371
+ 2024-01-12 02:39:08,304 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
372
+ 2024-01-12 02:39:09,305 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
373
+ 2024-01-12 02:39:11,877 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
374
+ 2024-01-12 02:39:14,306 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
375
+ 2024-01-12 02:39:16,880 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
376
+ 2024-01-12 02:39:19,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
377
+ 2024-01-12 02:39:21,881 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
378
+ 2024-01-12 02:39:24,308 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
379
+ 2024-01-12 02:39:26,882 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
380
+ 2024-01-12 02:39:29,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
381
+ 2024-01-12 02:39:31,884 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
382
+ 2024-01-12 02:39:34,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
383
+ 2024-01-12 02:39:36,885 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
384
+ 2024-01-12 02:39:38,305 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
385
+ 2024-01-12 02:39:40,306 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
386
+ 2024-01-12 02:39:41,890 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
387
+ 2024-01-12 02:39:45,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
388
+ 2024-01-12 02:39:46,892 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
389
+ 2024-01-12 02:39:50,308 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
390
+ 2024-01-12 02:39:51,893 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
391
+ 2024-01-12 02:39:55,309 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
392
+ 2024-01-12 02:39:56,894 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
393
+ 2024-01-12 02:40:00,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
394
+ 2024-01-12 02:40:01,895 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
395
+ 2024-01-12 02:40:05,310 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
396
+ 2024-01-12 02:40:06,896 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
397
+ 2024-01-12 02:40:08,306 DEBUG SenderThread:10874 [sender.py:send():382] send: stats
398
+ 2024-01-12 02:40:11,307 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
399
+ 2024-01-12 02:40:11,898 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
400
+ 2024-01-12 02:40:14,829 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: partial_history
401
+ 2024-01-12 02:40:14,831 DEBUG SenderThread:10874 [sender.py:send():382] send: history
402
+ 2024-01-12 02:40:14,831 DEBUG SenderThread:10874 [sender.py:send_request():409] send_request: summary_record
403
+ 2024-01-12 02:40:14,832 INFO SenderThread:10874 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
404
+ 2024-01-12 02:40:15,413 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/wandb-summary.json
405
+ 2024-01-12 02:40:16,413 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
406
+ 2024-01-12 02:40:16,832 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
407
+ 2024-01-12 02:40:17,119 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
408
+ 2024-01-12 02:40:18,414 INFO Thread-12 :10874 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240112_022837-twhkj0lf/files/output.log
409
+ 2024-01-12 02:40:22,122 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
410
+ 2024-01-12 02:40:22,588 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
411
+ 2024-01-12 02:40:27,125 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: keepalive
412
+ 2024-01-12 02:40:27,589 DEBUG HandlerThread:10874 [handler.py:handle_request():146] handle_request: status_report
wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb CHANGED
Binary files a/wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb and b/wandb/run-20240112_022837-twhkj0lf/run-twhkj0lf.wandb differ