Narkantak commited on
Commit
dc9b3d1
·
verified ·
1 Parent(s): d6c6dea

Narkantak/mistral-7b-Q-ashu-ft

Browse files
wandb/debug-internal.log CHANGED
@@ -619,3 +619,99 @@
619
  2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
620
  2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
621
  2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
619
  2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
620
  2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
621
  2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
622
+ 2024-03-14 07:41:41,707 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
623
+ 2024-03-14 07:41:41,936 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
624
+ 2024-03-14 07:41:41,936 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
625
+ 2024-03-14 07:41:41,937 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
626
+ 2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
627
+ 2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined disk monitor
628
+ 2024-03-14 07:41:41,940 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
629
+ 2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
630
+ 2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
631
+ 2024-03-14 07:41:41,955 INFO HandlerThread:508 [interfaces.py:finish():202] Joined gpu monitor
632
+ 2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined memory monitor
633
+ 2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined network monitor
634
+ 2024-03-14 07:41:41,956 DEBUG SenderThread:508 [sender.py:send():382] send: stats
635
+ 2024-03-14 07:41:46,449 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
636
+ 2024-03-14 07:41:46,957 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
637
+ 2024-03-14 07:41:51,450 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
638
+ 2024-03-14 07:41:51,958 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
639
+ 2024-03-14 07:41:56,451 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
640
+ 2024-03-14 07:41:56,959 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
641
+ 2024-03-14 07:42:01,453 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
642
+ 2024-03-14 07:42:01,960 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
643
+ 2024-03-14 07:42:06,454 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
644
+ 2024-03-14 07:42:06,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
645
+ 2024-03-14 07:42:11,456 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
646
+ 2024-03-14 07:42:11,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
647
+ 2024-03-14 07:42:16,457 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
648
+ 2024-03-14 07:42:16,963 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
649
+ 2024-03-14 07:42:21,458 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
650
+ 2024-03-14 07:42:21,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
651
+ 2024-03-14 07:42:26,459 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
652
+ 2024-03-14 07:42:26,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
653
+ 2024-03-14 07:42:31,460 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
654
+ 2024-03-14 07:42:31,965 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
655
+ 2024-03-14 07:42:36,462 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
656
+ 2024-03-14 07:42:36,966 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
657
+ 2024-03-14 07:42:41,463 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
658
+ 2024-03-14 07:42:41,967 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
659
+ 2024-03-14 07:42:46,464 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
660
+ 2024-03-14 07:42:46,968 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
661
+ 2024-03-14 07:42:51,465 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
662
+ 2024-03-14 07:42:51,969 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
663
+ 2024-03-14 07:42:56,466 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
664
+ 2024-03-14 07:42:56,970 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
665
+ 2024-03-14 07:43:01,467 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
666
+ 2024-03-14 07:43:01,971 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
667
+ 2024-03-14 07:43:06,469 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
668
+ 2024-03-14 07:43:06,972 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
669
+ 2024-03-14 07:43:11,470 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
670
+ 2024-03-14 07:43:11,973 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
671
+ 2024-03-14 07:43:16,471 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
672
+ 2024-03-14 07:43:16,974 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
673
+ 2024-03-14 07:43:21,472 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
674
+ 2024-03-14 07:43:21,975 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
675
+ 2024-03-14 07:43:26,473 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
676
+ 2024-03-14 07:43:26,976 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
677
+ 2024-03-14 07:43:31,475 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
678
+ 2024-03-14 07:43:31,977 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
679
+ 2024-03-14 07:43:36,476 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
680
+ 2024-03-14 07:43:36,978 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
681
+ 2024-03-14 07:43:41,477 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
682
+ 2024-03-14 07:43:41,979 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
683
+ 2024-03-14 07:43:46,478 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
684
+ 2024-03-14 07:43:46,980 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
685
+ 2024-03-14 07:43:51,479 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
686
+ 2024-03-14 07:43:51,981 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
687
+ 2024-03-14 07:43:56,480 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
688
+ 2024-03-14 07:43:56,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
689
+ 2024-03-14 07:44:01,482 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
690
+ 2024-03-14 07:44:01,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
691
+ 2024-03-14 07:44:06,483 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
692
+ 2024-03-14 07:44:06,867 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
693
+ 2024-03-14 07:44:06,867 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
694
+ 2024-03-14 07:44:06,867 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
695
+ 2024-03-14 07:44:06,868 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
696
+ 2024-03-14 07:44:06,869 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
697
+ 2024-03-14 07:44:06,869 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
698
+ 2024-03-14 07:44:06,870 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
699
+ 2024-03-14 07:44:06,870 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
700
+ 2024-03-14 07:44:06,871 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
701
+ 2024-03-14 07:44:06,871 WARNING HandlerThread:508 [interfaces.py:finish():207] Failed to finish disk monitoring: cannot join thread before it is started
702
+ 2024-03-14 07:44:06,871 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
703
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
704
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
705
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
706
+ 2024-03-14 07:44:06,873 DEBUG SenderThread:508 [sender.py:send():382] send: stats
707
+ 2024-03-14 07:44:07,874 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
708
+ 2024-03-14 07:44:09,742 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
709
+ 2024-03-14 07:44:09,742 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
710
+ 2024-03-14 07:44:09,742 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
711
+ 2024-03-14 07:44:09,742 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
712
+ 2024-03-14 07:44:09,743 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
713
+ 2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
714
+ 2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started gpu monitoring
715
+ 2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
716
+ 2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
717
+ 2024-03-14 07:44:11,484 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
wandb/debug.log CHANGED
@@ -38,3 +38,9 @@ config: {}
38
  2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
39
  2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
40
  2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
 
 
 
 
 
 
 
38
  2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
39
  2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
40
  2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
41
+ 2024-03-14 07:41:41,935 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
42
+ 2024-03-14 07:41:41,936 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
43
+ 2024-03-14 07:44:06,866 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
44
+ 2024-03-14 07:44:06,868 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
45
+ 2024-03-14 07:44:06,868 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
46
+ 2024-03-14 07:44:09,741 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
wandb/run-20240314_072629-9blbc0lc/logs/debug-internal.log CHANGED
@@ -619,3 +619,99 @@
619
  2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
620
  2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
621
  2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
619
  2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
620
  2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
621
  2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
622
+ 2024-03-14 07:41:41,707 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
623
+ 2024-03-14 07:41:41,936 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
624
+ 2024-03-14 07:41:41,936 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
625
+ 2024-03-14 07:41:41,937 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
626
+ 2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
627
+ 2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined disk monitor
628
+ 2024-03-14 07:41:41,940 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
629
+ 2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
630
+ 2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
631
+ 2024-03-14 07:41:41,955 INFO HandlerThread:508 [interfaces.py:finish():202] Joined gpu monitor
632
+ 2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined memory monitor
633
+ 2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined network monitor
634
+ 2024-03-14 07:41:41,956 DEBUG SenderThread:508 [sender.py:send():382] send: stats
635
+ 2024-03-14 07:41:46,449 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
636
+ 2024-03-14 07:41:46,957 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
637
+ 2024-03-14 07:41:51,450 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
638
+ 2024-03-14 07:41:51,958 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
639
+ 2024-03-14 07:41:56,451 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
640
+ 2024-03-14 07:41:56,959 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
641
+ 2024-03-14 07:42:01,453 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
642
+ 2024-03-14 07:42:01,960 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
643
+ 2024-03-14 07:42:06,454 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
644
+ 2024-03-14 07:42:06,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
645
+ 2024-03-14 07:42:11,456 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
646
+ 2024-03-14 07:42:11,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
647
+ 2024-03-14 07:42:16,457 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
648
+ 2024-03-14 07:42:16,963 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
649
+ 2024-03-14 07:42:21,458 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
650
+ 2024-03-14 07:42:21,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
651
+ 2024-03-14 07:42:26,459 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
652
+ 2024-03-14 07:42:26,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
653
+ 2024-03-14 07:42:31,460 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
654
+ 2024-03-14 07:42:31,965 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
655
+ 2024-03-14 07:42:36,462 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
656
+ 2024-03-14 07:42:36,966 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
657
+ 2024-03-14 07:42:41,463 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
658
+ 2024-03-14 07:42:41,967 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
659
+ 2024-03-14 07:42:46,464 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
660
+ 2024-03-14 07:42:46,968 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
661
+ 2024-03-14 07:42:51,465 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
662
+ 2024-03-14 07:42:51,969 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
663
+ 2024-03-14 07:42:56,466 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
664
+ 2024-03-14 07:42:56,970 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
665
+ 2024-03-14 07:43:01,467 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
666
+ 2024-03-14 07:43:01,971 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
667
+ 2024-03-14 07:43:06,469 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
668
+ 2024-03-14 07:43:06,972 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
669
+ 2024-03-14 07:43:11,470 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
670
+ 2024-03-14 07:43:11,973 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
671
+ 2024-03-14 07:43:16,471 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
672
+ 2024-03-14 07:43:16,974 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
673
+ 2024-03-14 07:43:21,472 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
674
+ 2024-03-14 07:43:21,975 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
675
+ 2024-03-14 07:43:26,473 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
676
+ 2024-03-14 07:43:26,976 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
677
+ 2024-03-14 07:43:31,475 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
678
+ 2024-03-14 07:43:31,977 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
679
+ 2024-03-14 07:43:36,476 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
680
+ 2024-03-14 07:43:36,978 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
681
+ 2024-03-14 07:43:41,477 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
682
+ 2024-03-14 07:43:41,979 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
683
+ 2024-03-14 07:43:46,478 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
684
+ 2024-03-14 07:43:46,980 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
685
+ 2024-03-14 07:43:51,479 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
686
+ 2024-03-14 07:43:51,981 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
687
+ 2024-03-14 07:43:56,480 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
688
+ 2024-03-14 07:43:56,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
689
+ 2024-03-14 07:44:01,482 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
690
+ 2024-03-14 07:44:01,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
691
+ 2024-03-14 07:44:06,483 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
692
+ 2024-03-14 07:44:06,867 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
693
+ 2024-03-14 07:44:06,867 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
694
+ 2024-03-14 07:44:06,867 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
695
+ 2024-03-14 07:44:06,868 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
696
+ 2024-03-14 07:44:06,869 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
697
+ 2024-03-14 07:44:06,869 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
698
+ 2024-03-14 07:44:06,870 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
699
+ 2024-03-14 07:44:06,870 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
700
+ 2024-03-14 07:44:06,871 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
701
+ 2024-03-14 07:44:06,871 WARNING HandlerThread:508 [interfaces.py:finish():207] Failed to finish disk monitoring: cannot join thread before it is started
702
+ 2024-03-14 07:44:06,871 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
703
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
704
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
705
+ 2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
706
+ 2024-03-14 07:44:06,873 DEBUG SenderThread:508 [sender.py:send():382] send: stats
707
+ 2024-03-14 07:44:07,874 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
708
+ 2024-03-14 07:44:09,742 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
709
+ 2024-03-14 07:44:09,742 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
710
+ 2024-03-14 07:44:09,742 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
711
+ 2024-03-14 07:44:09,742 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
712
+ 2024-03-14 07:44:09,743 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
713
+ 2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
714
+ 2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started gpu monitoring
715
+ 2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
716
+ 2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
717
+ 2024-03-14 07:44:11,484 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
wandb/run-20240314_072629-9blbc0lc/logs/debug.log CHANGED
@@ -38,3 +38,9 @@ config: {}
38
  2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
39
  2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
40
  2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
 
 
 
 
 
 
 
38
  2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
39
  2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
40
  2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
41
+ 2024-03-14 07:41:41,935 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
42
+ 2024-03-14 07:41:41,936 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
43
+ 2024-03-14 07:44:06,866 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
44
+ 2024-03-14 07:44:06,868 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
45
+ 2024-03-14 07:44:06,868 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
46
+ 2024-03-14 07:44:09,741 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb CHANGED
Binary files a/wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb and b/wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb differ