Narkantak/mistral-7b-Q-ashu-ft
Browse files
wandb/debug-internal.log
CHANGED
|
@@ -619,3 +619,99 @@
|
|
| 619 |
2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 620 |
2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 621 |
2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 619 |
2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 620 |
2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 621 |
2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 622 |
+
2024-03-14 07:41:41,707 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 623 |
+
2024-03-14 07:41:41,936 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
|
| 624 |
+
2024-03-14 07:41:41,936 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
|
| 625 |
+
2024-03-14 07:41:41,937 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
|
| 626 |
+
2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
|
| 627 |
+
2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined disk monitor
|
| 628 |
+
2024-03-14 07:41:41,940 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
| 629 |
+
2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
| 630 |
+
2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
|
| 631 |
+
2024-03-14 07:41:41,955 INFO HandlerThread:508 [interfaces.py:finish():202] Joined gpu monitor
|
| 632 |
+
2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined memory monitor
|
| 633 |
+
2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined network monitor
|
| 634 |
+
2024-03-14 07:41:41,956 DEBUG SenderThread:508 [sender.py:send():382] send: stats
|
| 635 |
+
2024-03-14 07:41:46,449 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 636 |
+
2024-03-14 07:41:46,957 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 637 |
+
2024-03-14 07:41:51,450 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 638 |
+
2024-03-14 07:41:51,958 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 639 |
+
2024-03-14 07:41:56,451 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 640 |
+
2024-03-14 07:41:56,959 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 641 |
+
2024-03-14 07:42:01,453 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 642 |
+
2024-03-14 07:42:01,960 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 643 |
+
2024-03-14 07:42:06,454 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 644 |
+
2024-03-14 07:42:06,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 645 |
+
2024-03-14 07:42:11,456 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 646 |
+
2024-03-14 07:42:11,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 647 |
+
2024-03-14 07:42:16,457 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 648 |
+
2024-03-14 07:42:16,963 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 649 |
+
2024-03-14 07:42:21,458 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 650 |
+
2024-03-14 07:42:21,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 651 |
+
2024-03-14 07:42:26,459 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 652 |
+
2024-03-14 07:42:26,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 653 |
+
2024-03-14 07:42:31,460 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 654 |
+
2024-03-14 07:42:31,965 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 655 |
+
2024-03-14 07:42:36,462 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 656 |
+
2024-03-14 07:42:36,966 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 657 |
+
2024-03-14 07:42:41,463 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 658 |
+
2024-03-14 07:42:41,967 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 659 |
+
2024-03-14 07:42:46,464 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 660 |
+
2024-03-14 07:42:46,968 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 661 |
+
2024-03-14 07:42:51,465 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 662 |
+
2024-03-14 07:42:51,969 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 663 |
+
2024-03-14 07:42:56,466 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 664 |
+
2024-03-14 07:42:56,970 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 665 |
+
2024-03-14 07:43:01,467 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 666 |
+
2024-03-14 07:43:01,971 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 667 |
+
2024-03-14 07:43:06,469 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 668 |
+
2024-03-14 07:43:06,972 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 669 |
+
2024-03-14 07:43:11,470 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 670 |
+
2024-03-14 07:43:11,973 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 671 |
+
2024-03-14 07:43:16,471 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 672 |
+
2024-03-14 07:43:16,974 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 673 |
+
2024-03-14 07:43:21,472 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 674 |
+
2024-03-14 07:43:21,975 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 675 |
+
2024-03-14 07:43:26,473 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 676 |
+
2024-03-14 07:43:26,976 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 677 |
+
2024-03-14 07:43:31,475 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 678 |
+
2024-03-14 07:43:31,977 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 679 |
+
2024-03-14 07:43:36,476 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 680 |
+
2024-03-14 07:43:36,978 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 681 |
+
2024-03-14 07:43:41,477 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 682 |
+
2024-03-14 07:43:41,979 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 683 |
+
2024-03-14 07:43:46,478 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 684 |
+
2024-03-14 07:43:46,980 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 685 |
+
2024-03-14 07:43:51,479 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 686 |
+
2024-03-14 07:43:51,981 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 687 |
+
2024-03-14 07:43:56,480 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 688 |
+
2024-03-14 07:43:56,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 689 |
+
2024-03-14 07:44:01,482 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 690 |
+
2024-03-14 07:44:01,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 691 |
+
2024-03-14 07:44:06,483 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 692 |
+
2024-03-14 07:44:06,867 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
|
| 693 |
+
2024-03-14 07:44:06,867 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
|
| 694 |
+
2024-03-14 07:44:06,867 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
|
| 695 |
+
2024-03-14 07:44:06,868 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
| 696 |
+
2024-03-14 07:44:06,869 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
|
| 697 |
+
2024-03-14 07:44:06,869 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
|
| 698 |
+
2024-03-14 07:44:06,870 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
|
| 699 |
+
2024-03-14 07:44:06,870 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
|
| 700 |
+
2024-03-14 07:44:06,871 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
|
| 701 |
+
2024-03-14 07:44:06,871 WARNING HandlerThread:508 [interfaces.py:finish():207] Failed to finish disk monitoring: cannot join thread before it is started
|
| 702 |
+
2024-03-14 07:44:06,871 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
|
| 703 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
| 704 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
| 705 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
|
| 706 |
+
2024-03-14 07:44:06,873 DEBUG SenderThread:508 [sender.py:send():382] send: stats
|
| 707 |
+
2024-03-14 07:44:07,874 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 708 |
+
2024-03-14 07:44:09,742 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
|
| 709 |
+
2024-03-14 07:44:09,742 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
|
| 710 |
+
2024-03-14 07:44:09,742 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
|
| 711 |
+
2024-03-14 07:44:09,742 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
| 712 |
+
2024-03-14 07:44:09,743 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
|
| 713 |
+
2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
|
| 714 |
+
2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started gpu monitoring
|
| 715 |
+
2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 716 |
+
2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 717 |
+
2024-03-14 07:44:11,484 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
wandb/debug.log
CHANGED
|
@@ -38,3 +38,9 @@ config: {}
|
|
| 38 |
2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 39 |
2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 40 |
2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 39 |
2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 40 |
2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
| 41 |
+
2024-03-14 07:41:41,935 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 42 |
+
2024-03-14 07:41:41,936 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 43 |
+
2024-03-14 07:44:06,866 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
| 44 |
+
2024-03-14 07:44:06,868 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 45 |
+
2024-03-14 07:44:06,868 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 46 |
+
2024-03-14 07:44:09,741 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
wandb/run-20240314_072629-9blbc0lc/logs/debug-internal.log
CHANGED
|
@@ -619,3 +619,99 @@
|
|
| 619 |
2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 620 |
2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 621 |
2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 619 |
2024-03-14 07:41:38,513 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 620 |
2024-03-14 07:41:38,514 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 621 |
2024-03-14 07:41:41,448 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 622 |
+
2024-03-14 07:41:41,707 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 623 |
+
2024-03-14 07:41:41,936 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
|
| 624 |
+
2024-03-14 07:41:41,936 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
|
| 625 |
+
2024-03-14 07:41:41,937 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
|
| 626 |
+
2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
|
| 627 |
+
2024-03-14 07:41:41,940 INFO HandlerThread:508 [interfaces.py:finish():202] Joined disk monitor
|
| 628 |
+
2024-03-14 07:41:41,940 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
| 629 |
+
2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
| 630 |
+
2024-03-14 07:41:41,943 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
|
| 631 |
+
2024-03-14 07:41:41,955 INFO HandlerThread:508 [interfaces.py:finish():202] Joined gpu monitor
|
| 632 |
+
2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined memory monitor
|
| 633 |
+
2024-03-14 07:41:41,956 INFO HandlerThread:508 [interfaces.py:finish():202] Joined network monitor
|
| 634 |
+
2024-03-14 07:41:41,956 DEBUG SenderThread:508 [sender.py:send():382] send: stats
|
| 635 |
+
2024-03-14 07:41:46,449 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 636 |
+
2024-03-14 07:41:46,957 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 637 |
+
2024-03-14 07:41:51,450 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 638 |
+
2024-03-14 07:41:51,958 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 639 |
+
2024-03-14 07:41:56,451 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 640 |
+
2024-03-14 07:41:56,959 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 641 |
+
2024-03-14 07:42:01,453 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 642 |
+
2024-03-14 07:42:01,960 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 643 |
+
2024-03-14 07:42:06,454 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 644 |
+
2024-03-14 07:42:06,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 645 |
+
2024-03-14 07:42:11,456 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 646 |
+
2024-03-14 07:42:11,961 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 647 |
+
2024-03-14 07:42:16,457 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 648 |
+
2024-03-14 07:42:16,963 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 649 |
+
2024-03-14 07:42:21,458 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 650 |
+
2024-03-14 07:42:21,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 651 |
+
2024-03-14 07:42:26,459 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 652 |
+
2024-03-14 07:42:26,964 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 653 |
+
2024-03-14 07:42:31,460 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 654 |
+
2024-03-14 07:42:31,965 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 655 |
+
2024-03-14 07:42:36,462 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 656 |
+
2024-03-14 07:42:36,966 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 657 |
+
2024-03-14 07:42:41,463 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 658 |
+
2024-03-14 07:42:41,967 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 659 |
+
2024-03-14 07:42:46,464 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 660 |
+
2024-03-14 07:42:46,968 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 661 |
+
2024-03-14 07:42:51,465 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 662 |
+
2024-03-14 07:42:51,969 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 663 |
+
2024-03-14 07:42:56,466 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 664 |
+
2024-03-14 07:42:56,970 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 665 |
+
2024-03-14 07:43:01,467 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 666 |
+
2024-03-14 07:43:01,971 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 667 |
+
2024-03-14 07:43:06,469 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 668 |
+
2024-03-14 07:43:06,972 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 669 |
+
2024-03-14 07:43:11,470 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 670 |
+
2024-03-14 07:43:11,973 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 671 |
+
2024-03-14 07:43:16,471 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 672 |
+
2024-03-14 07:43:16,974 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 673 |
+
2024-03-14 07:43:21,472 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 674 |
+
2024-03-14 07:43:21,975 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 675 |
+
2024-03-14 07:43:26,473 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 676 |
+
2024-03-14 07:43:26,976 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 677 |
+
2024-03-14 07:43:31,475 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 678 |
+
2024-03-14 07:43:31,977 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 679 |
+
2024-03-14 07:43:36,476 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 680 |
+
2024-03-14 07:43:36,978 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 681 |
+
2024-03-14 07:43:41,477 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 682 |
+
2024-03-14 07:43:41,979 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 683 |
+
2024-03-14 07:43:46,478 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 684 |
+
2024-03-14 07:43:46,980 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 685 |
+
2024-03-14 07:43:51,479 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 686 |
+
2024-03-14 07:43:51,981 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 687 |
+
2024-03-14 07:43:56,480 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 688 |
+
2024-03-14 07:43:56,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 689 |
+
2024-03-14 07:44:01,482 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 690 |
+
2024-03-14 07:44:01,982 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 691 |
+
2024-03-14 07:44:06,483 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
| 692 |
+
2024-03-14 07:44:06,867 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
|
| 693 |
+
2024-03-14 07:44:06,867 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
|
| 694 |
+
2024-03-14 07:44:06,867 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
|
| 695 |
+
2024-03-14 07:44:06,868 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
| 696 |
+
2024-03-14 07:44:06,869 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: pause
|
| 697 |
+
2024-03-14 07:44:06,869 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
|
| 698 |
+
2024-03-14 07:44:06,870 INFO HandlerThread:508 [handler.py:handle_request_pause():708] stopping system metrics thread
|
| 699 |
+
2024-03-14 07:44:06,870 INFO HandlerThread:508 [system_monitor.py:finish():203] Stopping system monitor
|
| 700 |
+
2024-03-14 07:44:06,871 INFO HandlerThread:508 [interfaces.py:finish():202] Joined cpu monitor
|
| 701 |
+
2024-03-14 07:44:06,871 WARNING HandlerThread:508 [interfaces.py:finish():207] Failed to finish disk monitoring: cannot join thread before it is started
|
| 702 |
+
2024-03-14 07:44:06,871 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
|
| 703 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():172] Starting system metrics aggregation loop
|
| 704 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():179] Finished system metrics aggregation loop
|
| 705 |
+
2024-03-14 07:44:06,871 DEBUG SystemMonitor:508 [system_monitor.py:_start():183] Publishing last batch of metrics
|
| 706 |
+
2024-03-14 07:44:06,873 DEBUG SenderThread:508 [sender.py:send():382] send: stats
|
| 707 |
+
2024-03-14 07:44:07,874 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: status_report
|
| 708 |
+
2024-03-14 07:44:09,742 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: resume
|
| 709 |
+
2024-03-14 07:44:09,742 INFO HandlerThread:508 [handler.py:handle_request_resume():699] starting system metrics thread
|
| 710 |
+
2024-03-14 07:44:09,742 INFO HandlerThread:508 [system_monitor.py:start():194] Starting system monitor
|
| 711 |
+
2024-03-14 07:44:09,742 INFO SystemMonitor:508 [system_monitor.py:_start():158] Starting system asset monitoring threads
|
| 712 |
+
2024-03-14 07:44:09,743 INFO SystemMonitor:508 [interfaces.py:start():190] Started cpu monitoring
|
| 713 |
+
2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started disk monitoring
|
| 714 |
+
2024-03-14 07:44:09,744 INFO SystemMonitor:508 [interfaces.py:start():190] Started gpu monitoring
|
| 715 |
+
2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started memory monitoring
|
| 716 |
+
2024-03-14 07:44:09,746 INFO SystemMonitor:508 [interfaces.py:start():190] Started network monitoring
|
| 717 |
+
2024-03-14 07:44:11,484 DEBUG HandlerThread:508 [handler.py:handle_request():146] handle_request: keepalive
|
wandb/run-20240314_072629-9blbc0lc/logs/debug.log
CHANGED
|
@@ -38,3 +38,9 @@ config: {}
|
|
| 38 |
2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 39 |
2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 40 |
2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
2024-03-14 07:40:55,669 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 39 |
2024-03-14 07:40:55,669 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 40 |
2024-03-14 07:41:38,508 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
| 41 |
+
2024-03-14 07:41:41,935 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 42 |
+
2024-03-14 07:41:41,936 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 43 |
+
2024-03-14 07:44:06,866 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
| 44 |
+
2024-03-14 07:44:06,868 INFO MainThread:34 [jupyter.py:save_ipynb():373] not saving jupyter notebook
|
| 45 |
+
2024-03-14 07:44:06,868 INFO MainThread:34 [wandb_init.py:_pause_backend():437] pausing backend
|
| 46 |
+
2024-03-14 07:44:09,741 INFO MainThread:34 [wandb_init.py:_resume_backend():442] resuming backend
|
wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb
CHANGED
|
Binary files a/wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb and b/wandb/run-20240314_072629-9blbc0lc/run-9blbc0lc.wandb differ
|
|
|