sharukat commited on
Commit
426eaab
·
verified ·
1 Parent(s): aa5ce0c

Training in progress, epoch 9

Browse files
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:eb455317e6e44818cd5895c227773b809a3117630c09ddc603402ced97e4add0
3
  size 502675828
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3949776af651054df791f80cf6017ee70cdc02dd3b2f95d7d62e238637ef9d42
3
  size 502675828
runs/Mar06_15-06-49_41759fa8e6ad/events.out.tfevents.1709737609.41759fa8e6ad.34.4 CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0cb13f81c94f361a52ad50c0947a690717d63f5cfbe2275abd652e9e02fff9a1
3
- size 10582
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:70ac2f1b6daf057bd8b581a174e30c46f398e3067a9c57f300c8be4f8185c8d9
3
+ size 11265
wandb/debug-internal.log CHANGED
@@ -928,3 +928,34 @@
928
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
929
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
930
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
928
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
929
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
930
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
931
+ 2024-03-06 15:11:26,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
932
+ 2024-03-06 15:11:27,806 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
933
+ 2024-03-06 15:11:29,033 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
934
+ 2024-03-06 15:11:30,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
935
+ 2024-03-06 15:11:32,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
936
+ 2024-03-06 15:11:34,034 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
937
+ 2024-03-06 15:11:35,383 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
938
+ 2024-03-06 15:11:37,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
939
+ 2024-03-06 15:11:39,035 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
940
+ 2024-03-06 15:11:39,899 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
941
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send():382] send: history
942
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
943
+ 2024-03-06 15:11:39,907 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
944
+ 2024-03-06 15:11:40,150 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
945
+ 2024-03-06 15:11:40,384 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
946
+ 2024-03-06 15:11:41,013 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
947
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send():382] send: history
948
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
949
+ 2024-03-06 15:11:41,016 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
950
+ 2024-03-06 15:11:41,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
951
+ 2024-03-06 15:11:42,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/output.log
952
+ 2024-03-06 15:11:43,017 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
953
+ 2024-03-06 15:11:44,048 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
954
+ 2024-03-06 15:11:45,385 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
955
+ 2024-03-06 15:11:48,018 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
956
+ 2024-03-06 15:11:49,049 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
957
+ 2024-03-06 15:11:50,386 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
958
+ 2024-03-06 15:11:53,019 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
959
+ 2024-03-06 15:11:54,051 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
960
+ 2024-03-06 15:11:55,387 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
961
+ 2024-03-06 15:11:56,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
wandb/run-20240306_145424-trm7fvg4/logs/debug-internal.log CHANGED
@@ -979,3 +979,34 @@ wandb.errors.AuthenticationError: The API key you provided is either invalid or
979
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
980
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
981
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
979
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
980
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
981
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
982
+ 2024-03-06 15:11:26,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
983
+ 2024-03-06 15:11:27,806 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
984
+ 2024-03-06 15:11:29,033 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
985
+ 2024-03-06 15:11:30,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
986
+ 2024-03-06 15:11:32,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
987
+ 2024-03-06 15:11:34,034 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
988
+ 2024-03-06 15:11:35,383 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
989
+ 2024-03-06 15:11:37,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
990
+ 2024-03-06 15:11:39,035 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
991
+ 2024-03-06 15:11:39,899 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
992
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send():382] send: history
993
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
994
+ 2024-03-06 15:11:39,907 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
995
+ 2024-03-06 15:11:40,150 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
996
+ 2024-03-06 15:11:40,384 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
997
+ 2024-03-06 15:11:41,013 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
998
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send():382] send: history
999
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
1000
+ 2024-03-06 15:11:41,016 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
1001
+ 2024-03-06 15:11:41,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
1002
+ 2024-03-06 15:11:42,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/output.log
1003
+ 2024-03-06 15:11:43,017 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1004
+ 2024-03-06 15:11:44,048 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
1005
+ 2024-03-06 15:11:45,385 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1006
+ 2024-03-06 15:11:48,018 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1007
+ 2024-03-06 15:11:49,049 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
1008
+ 2024-03-06 15:11:50,386 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1009
+ 2024-03-06 15:11:53,019 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1010
+ 2024-03-06 15:11:54,051 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
1011
+ 2024-03-06 15:11:55,387 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
1012
+ 2024-03-06 15:11:56,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
wandb/run-20240306_145455-h1uv5tyi/files/output.log CHANGED
@@ -54,3 +54,5 @@ Checkpoint destination directory /kaggle/working/checkpoint-186 already exists a
54
  _warn_prf(average, modifier, msg_start, len(result))
55
  /opt/conda/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
56
  _warn_prf(average, modifier, msg_start, len(result))
 
 
 
54
  _warn_prf(average, modifier, msg_start, len(result))
55
  /opt/conda/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
56
  _warn_prf(average, modifier, msg_start, len(result))
57
+ Checkpoint destination directory /kaggle/working/checkpoint-248 already exists and is non-empty. Saving will proceed but saved results may be invalid.
58
+ /opt/conda/lib/python3.10/site-packages/sklearn/metrics/_classification.py:1344: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 in labels with no predicted samples. Use `zero_division` parameter to control this behavior.
wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json CHANGED
@@ -1 +1 @@
1
- {"train/loss": 1.4226, "train/grad_norm": 6.0385541915893555, "train/learning_rate": 6.096774193548387e-06, "train/epoch": 8.0, "train/global_step": 248, "_timestamp": 1709737868.9472039, "_runtime": 973.2284469604492, "_step": 41, "eval/loss": 1.7184381484985352, "eval/accuracy": 0.41818181818181815, "eval/precision": 0.37482517482517486, "eval/recall": 0.41818181818181815, "eval/f1": 0.3393939393939394, "eval/runtime": 1.1132, "eval/samples_per_second": 49.406, "eval/steps_per_second": 3.593, "train/train_runtime": 237.4526, "train/train_samples_per_second": 10.339, "train/train_steps_per_second": 1.306, "train/total_flos": 645966638976000.0, "train/train_loss": 1.7031736066264491}
 
1
+ {"train/loss": 1.3877, "train/grad_norm": 6.286146640777588, "train/learning_rate": 3.0967741935483874e-06, "train/epoch": 9.0, "train/global_step": 279, "_timestamp": 1709737901.0131743, "_runtime": 1005.2944173812866, "_step": 43, "eval/loss": 1.7284780740737915, "eval/accuracy": 0.4, "eval/precision": 0.3380885780885781, "eval/recall": 0.4, "eval/f1": 0.3232677875938065, "eval/runtime": 1.1106, "eval/samples_per_second": 49.524, "eval/steps_per_second": 3.602, "train/train_runtime": 237.4526, "train/train_samples_per_second": 10.339, "train/train_steps_per_second": 1.306, "train/total_flos": 645966638976000.0, "train/train_loss": 1.7031736066264491}
wandb/run-20240306_145455-h1uv5tyi/logs/debug-internal.log CHANGED
@@ -928,3 +928,34 @@
928
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
929
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
930
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
928
  2024-03-06 15:11:21,953 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
929
  2024-03-06 15:11:24,032 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
930
  2024-03-06 15:11:25,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
931
+ 2024-03-06 15:11:26,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
932
+ 2024-03-06 15:11:27,806 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
933
+ 2024-03-06 15:11:29,033 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
934
+ 2024-03-06 15:11:30,382 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
935
+ 2024-03-06 15:11:32,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
936
+ 2024-03-06 15:11:34,034 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
937
+ 2024-03-06 15:11:35,383 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
938
+ 2024-03-06 15:11:37,807 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
939
+ 2024-03-06 15:11:39,035 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
940
+ 2024-03-06 15:11:39,899 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
941
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send():382] send: history
942
+ 2024-03-06 15:11:39,906 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
943
+ 2024-03-06 15:11:39,907 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
944
+ 2024-03-06 15:11:40,150 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
945
+ 2024-03-06 15:11:40,384 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
946
+ 2024-03-06 15:11:41,013 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: partial_history
947
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send():382] send: history
948
+ 2024-03-06 15:11:41,015 DEBUG SenderThread:137 [sender.py:send_request():409] send_request: summary_record
949
+ 2024-03-06 15:11:41,016 INFO SenderThread:137 [sender.py:_save_file():1403] saving file wandb-summary.json with policy end
950
+ 2024-03-06 15:11:41,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/wandb-summary.json
951
+ 2024-03-06 15:11:42,151 INFO Thread-18 :137 [dir_watcher.py:_on_file_modified():288] file/dir modified: /kaggle/working/wandb/run-20240306_145455-h1uv5tyi/files/output.log
952
+ 2024-03-06 15:11:43,017 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
953
+ 2024-03-06 15:11:44,048 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
954
+ 2024-03-06 15:11:45,385 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
955
+ 2024-03-06 15:11:48,018 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
956
+ 2024-03-06 15:11:49,049 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
957
+ 2024-03-06 15:11:50,386 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
958
+ 2024-03-06 15:11:53,019 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
959
+ 2024-03-06 15:11:54,051 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: keepalive
960
+ 2024-03-06 15:11:55,387 DEBUG HandlerThread:137 [handler.py:handle_request():146] handle_request: status_report
961
+ 2024-03-06 15:11:56,805 DEBUG SenderThread:137 [sender.py:send():382] send: stats
wandb/run-20240306_145455-h1uv5tyi/run-h1uv5tyi.wandb CHANGED
Binary files a/wandb/run-20240306_145455-h1uv5tyi/run-h1uv5tyi.wandb and b/wandb/run-20240306_145455-h1uv5tyi/run-h1uv5tyi.wandb differ