2024-07-03 23:21:13,488 INFO StreamThr :152382 [internal.py:wandb_internal():85] W&B internal server running at pid: 152382, started at: 2024-07-03 23:21:13.486632 2024-07-03 23:21:13,490 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status 2024-07-03 23:21:13,493 INFO WriterThread:152382 [datastore.py:open_for_write():87] open: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/run-0z9lks5c.wandb 2024-07-03 23:21:13,495 DEBUG SenderThread:152382 [sender.py:send():379] send: header 2024-07-03 23:21:13,568 DEBUG SenderThread:152382 [sender.py:send():379] send: run 2024-07-03 23:21:14,024 INFO SenderThread:152382 [dir_watcher.py:__init__():211] watching files in: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files 2024-07-03 23:21:14,024 INFO SenderThread:152382 [sender.py:_start_run_threads():1188] run started: 0z9lks5c with start time 1720041673.489436 2024-07-03 23:21:14,040 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: check_version 2024-07-03 23:21:14,040 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: check_version 2024-07-03 23:21:14,124 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: run_start 2024-07-03 23:21:14,153 DEBUG HandlerThread:152382 [system_info.py:__init__():26] System info init 2024-07-03 23:21:14,153 DEBUG HandlerThread:152382 [system_info.py:__init__():41] System info init done 2024-07-03 23:21:14,153 INFO HandlerThread:152382 [system_monitor.py:start():194] Starting system monitor 2024-07-03 23:21:14,153 INFO SystemMonitor:152382 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-07-03 23:21:14,154 INFO HandlerThread:152382 [system_monitor.py:probe():214] Collecting system info 2024-07-03 23:21:14,154 INFO SystemMonitor:152382 [interfaces.py:start():188] Started cpu monitoring 2024-07-03 23:21:14,155 INFO SystemMonitor:152382 [interfaces.py:start():188] Started disk monitoring 2024-07-03 23:21:14,155 INFO SystemMonitor:152382 [interfaces.py:start():188] Started gpu monitoring 2024-07-03 23:21:14,156 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,156 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,156 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,157 INFO SystemMonitor:152382 [interfaces.py:start():188] Started memory monitoring 2024-07-03 23:21:14,157 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,157 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,157 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,158 INFO SystemMonitor:152382 [interfaces.py:start():188] Started network monitoring 2024-07-03 23:21:14,158 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:14,359 DEBUG HandlerThread:152382 [system_info.py:probe():152] Probing system 2024-07-03 23:21:14,368 DEBUG HandlerThread:152382 [system_info.py:_probe_git():137] Probing git 2024-07-03 23:21:14,408 DEBUG HandlerThread:152382 [system_info.py:_probe_git():145] Probing git done 2024-07-03 23:21:14,408 DEBUG HandlerThread:152382 [system_info.py:probe():200] Probing system done 2024-07-03 23:21:14,408 DEBUG HandlerThread:152382 [system_monitor.py:probe():223] {'os': 'Linux-4.18.0-553.5.1.el8_10.x86_64-x86_64-with-glibc2.28', 'python': '3.9.19', 'heartbeatAt': '2024-07-03T21:21:14.359969', 'startedAt': '2024-07-03T21:21:13.458856', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--dataset_name=linoyts/Tuxemon', '--dataloader_num_workers=1', '--resolution=512', '--center_crop', '--random_flip', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--max_train_steps=15000', '--learning_rate=1e-04', '--max_grad_norm=1.0', '--lr_scheduler=cosine', '--lr_warmup_steps=0', '--output_dir=/system/user/studentwork/mohammed/DM/pokemon/pokemon_results', '--push_to_hub', '--hub_model_id=A7MED24/pokemon-lora', '--report_to=wandb', '--checkpointing_steps=500', '--validation_prompt=Totoro', '--seed=1337'), 'state': 'running', 'program': '/system/user/studentwork/mohammed/DM/pokemon/train_text2image_lora.py', 'codePathLocal': None, 'git': {'remote': 'https://github.com/huggingface/diffusers', 'commit': '0368483b61ad690aef5b0d92611270868c0ce8ea'}, 'email': None, 'root': '/system/user/studentwork/mohammed/DM/pokemon/diffusers', 'host': 'student06', 'username': 'mohammed', 'executable': '/system/user/mohammed/miniconda3/envs/dm/bin/python', 'cpu_count': 12, 'cpu_count_logical': 24, 'cpu_freq': {'current': 3508.261521739129, 'min': 2200.0, 'max': 3500.0}, 'cpu_freq_per_core': [{'current': 3211.586, 'min': 2200.0, 'max': 3500.0}, {'current': 3288.24, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.007, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.56, 'min': 2200.0, 'max': 3500.0}, {'current': 3416.434, 'min': 2200.0, 'max': 3500.0}, {'current': 3683.502, 'min': 2200.0, 'max': 3500.0}, {'current': 3520.499, 'min': 2200.0, 'max': 3500.0}, {'current': 3227.946, 'min': 2200.0, 'max': 3500.0}, {'current': 3540.521, 'min': 2200.0, 'max': 3500.0}, {'current': 3360.108, 'min': 2200.0, 'max': 3500.0}, {'current': 3698.971, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.56, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.56, 'min': 2200.0, 'max': 3500.0}, {'current': 3702.77, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.56, 'min': 2200.0, 'max': 3500.0}, {'current': 3760.436, 'min': 2200.0, 'max': 3500.0}, {'current': 3253.574, 'min': 2200.0, 'max': 3500.0}, {'current': 3241.464, 'min': 2200.0, 'max': 3500.0}, {'current': 3704.008, 'min': 2200.0, 'max': 3500.0}, {'current': 3699.559, 'min': 2200.0, 'max': 3500.0}, {'current': 3581.667, 'min': 2200.0, 'max': 3500.0}, {'current': 3237.474, 'min': 2200.0, 'max': 3500.0}, {'current': 3685.495, 'min': 2200.0, 'max': 3500.0}], 'disk': {'/': {'total': 200.01137161254883, 'used': 61.17613220214844}}, 'memory': {'total': 124.99904251098633}} 2024-07-03 23:21:14,408 INFO HandlerThread:152382 [system_monitor.py:probe():224] Finished collecting system info 2024-07-03 23:21:14,408 INFO HandlerThread:152382 [system_monitor.py:probe():227] Publishing system info 2024-07-03 23:21:14,408 DEBUG HandlerThread:152382 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2024-07-03 23:21:15,028 INFO Thread-12 :152382 [dir_watcher.py:_on_file_created():271] file/dir created: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/conda-environment.yaml 2024-07-03 23:21:16,158 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:16,985 DEBUG HandlerThread:152382 [system_info.py:_save_conda():224] Saving conda packages done 2024-07-03 23:21:16,989 INFO HandlerThread:152382 [system_monitor.py:probe():229] Finished publishing system info 2024-07-03 23:21:16,995 DEBUG SenderThread:152382 [sender.py:send():379] send: files 2024-07-03 23:21:16,995 INFO SenderThread:152382 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-07-03 23:21:17,029 INFO Thread-12 :152382 [dir_watcher.py:_on_file_modified():288] file/dir modified: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/conda-environment.yaml 2024-07-03 23:21:17,029 INFO Thread-12 :152382 [dir_watcher.py:_on_file_created():271] file/dir created: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/wandb-metadata.json 2024-07-03 23:21:17,191 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: python_packages 2024-07-03 23:21:17,191 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: python_packages 2024-07-03 23:21:17,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:21:17,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:17,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:21:17,450 DEBUG SenderThread:152382 [sender.py:send():379] send: telemetry 2024-07-03 23:21:17,451 DEBUG SenderThread:152382 [sender.py:send():379] send: config 2024-07-03 23:21:17,624 INFO wandb-upload_0:152382 [upload_job.py:push():130] Uploaded file /tmp/tmprsdfeoymwandb/41f7umyg-wandb-metadata.json 2024-07-03 23:21:18,030 INFO Thread-12 :152382 [dir_watcher.py:_on_file_created():271] file/dir created: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/requirements.txt 2024-07-03 23:21:18,031 INFO Thread-12 :152382 [dir_watcher.py:_on_file_created():271] file/dir created: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/output.log 2024-07-03 23:21:18,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,159 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:18,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:19,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:19,453 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:20,033 INFO Thread-12 :152382 [dir_watcher.py:_on_file_modified():288] file/dir modified: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/output.log 2024-07-03 23:21:20,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,160 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:20,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:21,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:22,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,161 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:22,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:23,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:24,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,162 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,163 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,163 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,163 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,163 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:24,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:25,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:25,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:26,163 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,164 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,164 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,164 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,164 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,164 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,165 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:26,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:27,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:28,165 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,165 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:28,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:29,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:30,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,166 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:30,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:31,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:31,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:32,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:21:32,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:21:32,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:33,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:34,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,167 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:34,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:35,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:36,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:36,466 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:37,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,168 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:38,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:39,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:40,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:41,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,169 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:42,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:42,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:43,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:44,169 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:44,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:45,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:46,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:47,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:21:47,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:21:47,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:47,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:48,054 INFO Thread-12 :152382 [dir_watcher.py:_on_file_modified():288] file/dir modified: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/config.yaml 2024-07-03 23:21:48,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,170 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:48,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:49,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:50,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:51,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,171 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:52,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:53,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:53,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:54,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:55,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:56,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:57,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:58,172 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:21:58,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:21:58,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:21:59,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:00,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:01,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,173 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:02,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:22:02,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:22:02,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:03,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:03,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:04,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:05,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,174 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:06,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:07,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:08,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:09,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:09,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:10,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:11,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:12,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,175 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:12,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:13,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:14,158 DEBUG SystemMonitor:152382 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-07-03 23:22:14,159 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:14,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:15,159 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:15,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:16,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,176 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:16,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:17,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:22:17,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:22:17,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:18,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:19,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,177 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:20,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:20,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:21,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:22,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:23,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:24,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:25,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:26,178 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:26,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:26,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:27,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:28,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:29,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,179 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:30,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:31,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:31,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:32,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:22:32,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:22:32,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:33,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,180 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:34,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:35,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:36,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:36,458 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:37,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:38,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:39,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:40,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,181 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:40,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:41,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:42,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:42,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:43,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:44,159 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,182 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:44,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:45,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:46,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:47,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:22:47,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:22:47,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:47,462 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,183 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:48,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:49,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:50,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:51,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:52,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:53,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:53,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:54,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,184 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:54,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:55,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:56,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:57,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,185 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:22:58,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:22:59,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:22:59,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:00,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:01,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:02,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:23:02,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:23:02,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:03,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:04,186 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:04,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:04,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:05,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:06,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:07,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:08,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,187 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:08,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:09,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:10,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:10,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:11,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,188 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:12,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:13,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:14,160 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:14,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:15,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:16,160 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:16,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:17,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:23:17,192 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:23:17,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:18,189 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:18,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:19,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:20,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:21,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:21,441 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:22,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,190 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:22,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:23,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:24,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:25,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,191 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:26,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:27,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:27,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:28,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:29,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:30,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:31,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:32,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:23:32,192 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:32,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:23:32,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:32,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:33,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:34,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:35,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:36,193 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:36,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:37,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:38,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:38,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:39,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:40,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,194 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:40,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:41,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:42,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:43,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:44,160 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:23:44,162 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:44,195 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:23:44,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,195 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:44,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:45,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:46,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:47,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:23:47,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:23:47,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,196 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:48,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:49,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:49,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:50,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:51,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:52,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:53,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:54,197 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:54,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:55,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:23:55,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:56,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:57,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:58,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,198 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:23:58,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:23:59,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:00,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:01,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:01,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:02,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:24:02,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,199 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:02,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:03,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:04,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:05,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,200 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:06,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:06,436 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:07,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:08,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:09,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:10,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:11,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:12,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:12,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,201 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:12,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:13,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:14,161 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:14,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:15,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:16,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,202 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:16,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:17,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:24:17,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:24:17,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:17,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:18,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:19,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,203 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:20,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:21,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:22,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:23,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:23,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:24,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:25,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:26,204 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:26,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:27,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:28,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:28,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:29,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:30,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,205 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:30,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:31,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:32,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:24:32,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:32,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:33,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:33,451 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,206 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:34,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:35,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:36,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:37,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:38,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:39,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:39,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:40,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,207 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:40,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:41,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:42,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:43,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:44,161 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,208 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:44,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:45,162 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:45,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:46,209 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:47,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:24:47,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:24:47,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:48,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:48,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:49,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:50,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:50,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:51,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:52,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:52,210 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:52,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:53,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:54,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:54,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:55,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:56,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:24:56,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,211 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:56,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:57,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:58,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:58,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:24:59,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:00,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:00,212 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:01,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:01,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:02,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:25:02,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:02,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:03,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:04,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:04,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:05,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:06,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:06,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,213 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:06,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:07,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:08,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:08,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:09,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:10,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,214 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:10,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:11,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:12,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:12,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:12,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:13,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:14,162 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:25:14,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:14,215 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:15,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:16,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:16,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:17,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:25:17,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:25:17,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:17,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:18,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:18,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:19,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:20,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:20,216 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:20,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:21,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:22,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:22,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:23,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:23,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:24,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:24,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,217 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:24,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:25,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:26,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:26,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:27,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:28,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:28,218 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:29,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:29,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:30,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:30,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:31,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:32,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:25:32,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:32,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:33,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:34,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:34,219 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:34,575 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:35,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:36,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:36,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:37,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:38,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:38,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,220 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:38,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:39,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:40,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:40,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:40,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:41,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:42,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:42,221 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:43,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:44,162 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:25:44,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:44,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:45,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:46,163 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:46,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:46,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:47,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:25:47,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:25:47,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:48,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:48,222 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:48,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:49,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:50,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:50,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:51,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:51,448 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:52,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:52,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,223 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:52,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:53,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:54,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:54,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:55,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:56,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:56,224 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:57,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:25:57,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:58,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:58,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:25:59,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:00,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:00,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:01,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:02,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:26:02,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:26:02,225 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:02,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:02,465 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:03,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:04,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:04,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:05,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:06,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:06,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,226 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:06,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:07,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:08,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:08,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:08,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:09,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:10,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:10,227 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:11,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:12,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:12,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:13,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:13,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:14,163 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:26:14,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:14,228 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:15,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:16,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:16,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:17,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:26:17,193 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:26:17,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:18,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:18,451 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:19,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:20,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:20,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,229 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:20,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:21,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:22,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:22,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:23,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:24,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:24,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:24,230 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:25,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:26,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:26,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:27,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:28,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:28,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:29,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:29,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:30,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:30,231 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:30,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:31,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:32,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:26:32,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:32,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:33,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:34,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:34,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,232 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:34,450 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:35,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:36,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:36,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:37,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:38,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:38,233 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:39,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:40,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:40,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:40,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:41,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:42,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:42,234 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:43,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:44,164 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:26:44,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:44,234 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:44,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:45,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:46,164 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:46,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:46,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:47,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:26:47,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:26:47,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:48,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:48,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,235 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:48,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:49,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:50,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:50,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:51,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:51,449 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:52,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,236 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:52,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:53,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:54,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:54,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:55,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:56,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:56,237 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:57,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:26:57,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:58,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:58,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:26:59,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:00,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:00,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:01,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:02,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:27:02,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:27:02,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:02,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,238 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:02,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:03,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:04,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:04,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:05,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:06,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:06,239 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:07,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:08,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:08,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:08,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:09,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:10,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:10,240 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:11,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:12,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:12,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:13,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:13,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:14,164 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:27:14,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:14,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:15,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:16,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:16,241 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:16,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:17,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:27:17,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:27:17,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:18,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:18,444 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:19,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:20,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:20,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,242 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:20,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:21,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:22,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:22,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:23,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:24,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:24,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:24,243 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:25,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:26,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:26,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:27,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:28,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:28,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:29,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:29,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:30,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:30,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,244 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:30,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:31,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:32,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:27:32,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:27:32,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:32,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:33,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:34,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,245 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:34,448 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:35,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:36,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:36,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:37,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:38,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:38,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:39,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:40,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:40,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:40,246 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:40,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:41,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:42,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:42,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:43,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:44,165 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:27:44,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:44,247 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:27:44,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,247 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:44,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:45,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:46,165 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:46,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:46,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:47,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:27:47,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:27:47,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:48,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:48,248 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:49,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:50,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:50,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:51,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:51,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:52,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:52,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:53,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:54,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:54,249 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:54,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:55,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:56,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:56,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:57,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:27:57,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:58,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:27:58,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,250 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:58,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:27:59,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:00,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:00,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:01,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:02,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:28:02,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:28:02,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,251 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:02,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:03,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:04,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:04,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:05,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:06,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:06,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:07,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:08,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:08,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:08,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,252 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:08,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:09,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:10,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:10,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:11,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:12,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:12,253 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:13,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:13,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:14,165 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:28:14,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:14,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:15,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:16,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:16,254 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:17,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:28:17,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:28:17,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:18,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:18,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:19,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:20,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:20,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:21,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:22,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:22,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,255 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:22,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:23,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:24,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:24,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:24,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:25,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:26,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:26,256 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:27,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:28,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:28,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:29,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:29,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:30,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:30,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:31,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:32,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:28:32,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:28:32,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:32,257 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:32,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:33,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:34,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:34,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:35,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:36,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:36,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,258 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:36,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:37,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:38,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:38,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:39,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:40,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:40,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:40,259 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:41,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:42,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:42,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:43,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:44,166 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:28:44,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:44,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:45,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:46,166 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:46,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:46,260 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:46,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:47,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:28:47,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:28:47,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:48,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:48,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:49,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:50,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:50,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,261 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:50,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:51,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:51,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:52,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:52,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:53,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:54,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:54,262 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:55,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:56,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:56,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:57,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:28:57,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:58,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:58,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:28:59,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:00,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:00,263 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:00,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:01,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:02,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:29:02,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:29:02,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:02,442 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:03,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:04,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,264 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:04,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:05,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:06,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:06,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:07,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:08,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:08,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:08,265 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:09,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:10,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:10,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:11,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:12,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:12,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:13,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:13,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:14,166 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:29:14,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:14,266 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:29:14,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,266 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:14,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:15,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:16,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:16,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:17,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:29:17,194 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:29:17,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:18,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,267 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:18,469 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:19,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:20,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:20,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:21,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:22,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:22,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:23,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:24,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:24,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:24,268 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:24,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:25,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:26,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:26,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:27,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:28,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:28,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,269 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:28,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:29,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:29,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:30,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:30,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:31,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:32,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:29:32,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:29:32,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:32,270 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:33,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:34,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:34,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:35,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:36,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:36,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:37,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:38,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:38,271 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:38,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:39,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:40,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:40,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:40,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:41,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:42,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,272 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:42,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:43,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:44,167 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:29:44,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:44,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:45,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:46,167 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:46,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:46,273 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:47,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:29:47,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:29:47,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:48,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:48,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:49,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:50,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:50,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:51,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:51,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:52,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:52,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,274 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:52,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:53,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:54,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:54,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:55,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:56,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:56,275 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:57,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:29:57,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:58,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:58,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:29:59,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:00,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:00,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:01,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:02,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:30:02,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:30:02,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:02,276 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:02,446 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:03,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:04,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:04,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:05,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:06,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:06,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,277 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:06,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:07,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:08,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:08,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:08,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:09,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:10,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:10,278 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:11,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:12,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:12,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:13,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:13,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:14,167 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:30:14,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:14,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:15,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:16,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:16,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,279 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:16,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:17,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:30:17,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:30:17,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:18,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:18,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:19,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:20,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,280 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:20,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:21,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:22,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:22,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:23,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:24,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:24,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:24,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:25,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:26,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:26,281 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:26,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:27,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:28,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:28,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:29,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:30,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:30,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:30,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,282 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:30,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:31,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:32,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:30:32,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:30:32,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:32,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:33,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:34,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:34,283 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:35,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:35,529 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:36,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:36,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:37,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:38,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:38,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:39,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:40,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:40,284 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:40,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:41,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:41,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:42,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:42,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:43,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:44,168 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:30:44,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:44,285 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:30:44,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,285 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:44,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:45,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:46,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:46,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:47,168 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:47,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:30:47,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:30:47,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:48,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:48,286 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:49,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:50,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:50,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:51,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:52,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:52,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:52,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:53,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:54,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:54,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,287 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:54,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:55,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:56,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:56,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:57,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:30:57,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:58,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,288 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:58,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:30:59,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:00,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:00,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:01,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:02,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:31:02,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:31:02,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:02,456 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:03,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:04,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:04,289 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:04,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:05,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:06,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:06,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:07,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:08,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:08,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:08,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,290 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:08,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:09,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:10,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:10,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:11,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:12,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:12,291 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:13,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:13,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:14,168 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:31:14,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:14,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:15,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:16,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:16,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:17,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:31:17,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:31:17,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:18,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:18,292 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:18,452 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:19,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:20,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:20,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:21,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:22,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:22,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,293 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:22,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:23,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:24,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:24,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:24,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:25,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:26,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:26,294 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:27,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:28,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:28,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:29,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:30,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:30,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:30,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:31,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:32,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:31:32,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:31:32,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:32,295 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:32,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:33,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:34,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:34,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:35,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:35,441 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:36,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,296 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:36,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:37,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:38,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:38,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:39,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:40,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:40,297 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:41,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:41,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:42,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:42,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:43,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:44,169 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:31:44,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:44,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:45,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:46,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:46,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,298 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:46,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:47,169 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:47,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:31:47,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:31:47,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:48,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:48,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:49,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:50,242 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:50,299 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:51,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:52,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:52,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:52,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:53,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:54,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:54,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:55,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:56,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:56,300 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:56,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:57,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:31:57,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:58,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:58,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:31:59,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:00,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:00,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,301 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:00,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:01,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:02,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:32:02,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:32:02,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:02,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:03,243 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:04,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:04,302 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:05,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:06,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:06,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:07,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:08,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:08,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:08,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:09,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:10,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:10,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,303 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:10,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:11,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:12,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:12,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:13,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:13,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:14,169 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:32:14,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,304 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:14,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:15,244 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:16,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:16,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:17,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:32:17,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:32:17,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:18,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,305 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:18,454 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:19,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:20,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:20,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:21,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:22,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:22,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:23,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:24,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:24,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:24,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,306 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:24,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:25,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:26,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:26,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:27,245 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:28,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:28,307 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:29,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:29,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:30,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:30,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:31,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:32,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:32:32,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:32:32,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:32,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:33,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:34,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:34,308 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:34,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:35,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:36,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:36,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:37,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:38,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:38,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,309 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:38,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:39,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:40,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:40,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:40,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:41,246 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:42,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:42,310 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:43,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:44,170 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:32:44,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:44,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:45,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:46,170 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:46,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:46,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:47,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:32:47,195 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:32:47,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:48,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:48,311 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:48,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:49,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:50,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:50,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:51,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:51,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:52,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,312 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:52,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:53,247 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:54,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:54,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:55,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:56,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:56,313 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:57,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:32:57,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:58,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:58,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:32:59,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:00,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:00,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:01,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:02,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:33:02,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:33:02,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:02,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,314 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:02,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:03,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:04,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:04,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:05,248 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:06,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:06,315 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:07,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:08,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:08,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:08,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:09,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:10,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:10,316 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:11,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:12,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:12,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:13,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:13,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:14,170 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:33:14,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:14,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:15,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:16,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:16,317 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:16,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:17,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:33:17,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:33:17,249 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:18,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:18,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:19,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:20,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,318 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:20,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:21,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:22,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:22,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:23,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:24,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:24,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:24,319 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:25,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:26,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:26,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:27,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:28,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:28,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:29,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:29,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:30,250 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:30,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,320 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:30,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:31,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:32,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:33:32,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:33:32,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:32,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:33,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:34,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,321 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:34,955 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:35,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:36,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:36,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:37,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:38,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:38,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:39,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:40,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:40,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:40,322 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:40,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:41,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:42,251 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:42,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:43,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:44,171 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:33:44,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,323 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:44,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:45,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:46,171 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:46,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:46,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:47,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:33:47,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:33:47,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:48,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:48,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:49,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:50,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:50,324 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:50,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:51,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:51,444 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:52,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:52,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:53,252 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:54,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:54,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,325 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:54,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:55,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:56,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:56,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:57,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:33:57,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:58,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:58,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:33:59,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:00,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:00,326 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:00,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:01,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:02,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:34:02,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:34:02,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:02,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:03,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:04,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:04,327 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:05,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:06,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:06,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:07,253 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:08,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:08,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:08,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:09,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:10,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:10,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,328 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:10,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:11,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:12,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:12,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:13,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:13,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:14,171 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:34:14,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,329 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:14,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:15,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:16,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:16,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:17,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:34:17,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:34:17,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:18,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:18,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:19,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:20,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:20,330 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:20,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:21,254 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:22,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:22,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:23,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:24,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:24,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:24,331 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:25,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:26,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:26,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:27,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:28,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:28,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:29,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:29,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:30,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:30,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,332 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:30,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:31,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:32,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:34:32,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:34:32,255 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:32,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:33,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:34,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:34,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:35,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:36,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:36,333 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:36,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:37,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:38,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:38,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:39,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:40,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:40,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:40,334 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:41,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:42,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:42,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:43,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:44,172 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:34:44,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:44,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:45,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:46,172 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:46,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:46,335 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:46,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:47,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:34:47,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:34:47,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:48,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:48,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:49,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:50,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:50,336 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:51,256 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:51,444 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:52,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:52,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:53,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:54,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:54,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:55,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:56,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:56,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,337 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:56,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:57,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:34:57,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:58,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:58,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:34:59,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:00,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:00,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:01,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:02,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:35:02,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:35:02,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:02,338 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:02,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:03,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:03,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:04,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:04,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:05,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:06,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:06,339 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:07,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:08,257 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:08,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:09,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:09,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:10,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:10,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:11,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:12,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:12,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,340 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:12,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:13,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:14,172 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:35:14,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:14,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:15,173 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:15,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:16,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:16,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:17,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:35:17,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:35:17,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:18,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:18,341 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:18,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:19,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:20,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:20,446 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:21,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:22,258 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:22,342 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:23,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:24,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:24,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:25,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:26,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:26,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:26,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:27,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:28,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:28,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,343 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:28,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:29,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:30,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:30,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:31,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:31,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:32,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:35:32,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:35:32,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:32,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:33,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:34,259 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:34,344 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:34,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:35,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:36,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:36,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:37,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:38,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:38,345 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:39,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:40,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:40,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:41,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:42,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:42,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:42,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:43,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:44,173 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:35:44,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:44,346 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:35:44,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,346 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:44,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:45,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:46,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:46,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:47,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:35:47,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:35:47,260 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:47,442 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:48,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:48,347 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:49,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:50,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:50,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:51,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:52,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:52,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:53,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:53,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:54,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,348 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:54,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:55,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:56,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:56,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:57,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:58,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:35:58,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:58,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:35:59,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:00,261 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:00,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,349 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:00,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:01,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:02,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:36:02,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:36:02,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:02,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:03,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:03,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:04,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:04,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:05,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:06,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:06,350 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:06,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:07,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:08,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:08,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:09,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:09,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:10,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:10,351 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:11,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:12,262 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:12,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:13,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:14,175 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:36:14,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:14,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:15,175 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:15,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:16,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:16,352 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:16,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:17,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:36:17,196 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:36:17,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:18,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:18,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:19,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:20,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,353 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:20,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:21,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:22,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:22,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:23,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:24,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:24,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:25,263 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:26,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:26,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:26,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,354 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:26,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:27,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:28,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:28,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:29,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:30,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:30,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:31,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:31,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:32,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:36:32,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:36:32,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:32,355 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:32,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:33,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:34,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:34,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:35,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:36,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,356 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:36,461 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:37,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:38,264 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:38,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:39,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:40,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:40,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:41,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:42,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:42,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,357 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:42,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:43,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:44,174 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:36:44,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:44,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:45,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:46,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:46,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:47,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:36:47,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:36:47,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:47,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:48,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:48,358 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:48,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:49,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:50,265 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:50,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:51,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:52,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:52,359 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:53,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:53,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:54,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:54,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:55,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:56,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:56,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:57,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:58,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:36:58,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:36:58,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,360 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:58,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:36:59,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:00,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:00,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:01,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:02,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:37:02,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:37:02,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:02,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:03,266 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:03,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:04,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:04,361 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:04,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:05,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:06,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:06,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:07,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:08,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:08,362 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:09,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:09,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:10,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:10,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:11,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:12,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:12,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:13,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:14,175 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:37:14,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:14,363 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:37:14,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,363 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:14,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:15,176 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:15,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:16,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:16,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:17,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:37:17,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:37:17,267 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:18,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:18,364 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:19,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:20,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:20,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:21,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:22,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:22,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:23,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:24,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:24,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,365 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:24,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:25,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:26,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:26,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:26,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:27,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:28,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:28,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:29,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:30,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:30,366 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:30,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:31,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:32,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:37:32,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:37:32,268 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:32,446 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:33,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:34,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:34,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,367 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:34,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:35,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:36,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:36,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:37,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:38,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:38,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:38,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:39,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:40,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:40,368 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:40,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:41,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:42,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:42,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:43,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:44,175 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:37:44,176 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:44,269 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,369 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:44,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:45,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:46,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:46,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:47,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:37:47,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:37:47,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:48,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:48,370 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:49,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:49,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:50,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:50,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:51,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:52,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:52,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:53,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:54,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:54,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,371 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:54,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:55,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:37:55,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:56,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:56,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:57,270 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:58,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:58,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:37:59,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:00,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:00,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:00,372 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:00,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:01,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:02,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:38:02,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:38:02,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:02,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:03,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:04,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,373 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:04,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:05,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:05,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:06,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:06,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:07,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:08,271 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:08,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:09,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:10,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:10,374 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:10,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:11,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:11,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:12,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:12,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:13,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:14,175 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:38:14,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,375 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:14,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:15,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:16,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:16,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:17,176 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:17,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:38:17,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:38:17,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:18,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:18,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:19,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:20,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:20,376 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:20,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:21,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:22,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:22,272 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:22,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:23,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:24,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:24,377 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:25,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:26,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:26,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:27,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:27,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:28,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:28,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:29,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:30,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:30,378 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:30,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:31,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:32,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:38:32,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:38:32,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:32,449 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:33,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:34,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:34,379 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:35,273 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:36,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:36,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:37,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:38,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:38,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:38,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:39,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:40,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:40,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,380 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:40,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:41,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:42,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:42,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:43,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:43,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:44,176 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:38:44,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:44,381 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:45,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:46,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:46,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:47,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:38:47,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:38:47,274 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:48,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:48,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:49,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:50,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:50,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,382 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:50,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:51,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:52,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:52,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:53,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:54,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:54,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:54,383 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:55,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:56,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:56,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:57,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:58,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:58,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:38:59,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:38:59,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:00,275 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:00,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,384 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:00,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:01,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:02,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:39:02,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:39:02,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:02,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:03,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:04,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,385 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:04,446 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:05,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:06,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:06,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:07,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:08,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:08,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:09,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:10,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:10,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:10,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,386 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:10,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:11,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:12,276 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:12,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:13,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:14,176 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:39:14,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:14,387 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:15,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:16,177 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:16,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:16,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:17,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:39:17,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:39:17,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:18,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:18,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:19,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:20,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:20,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,388 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:20,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:21,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:21,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:22,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:22,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:23,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:24,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:24,389 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:25,277 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:26,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:26,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:27,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:27,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:28,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:28,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:29,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:30,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:30,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,390 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:30,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:31,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:32,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:39:32,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:39:32,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:32,453 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:33,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:34,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:34,391 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:35,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:36,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:36,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:37,278 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:38,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:38,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:38,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:39,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:40,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:40,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,392 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:40,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:41,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:42,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:42,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:43,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:44,177 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:39:44,178 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:44,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:44,393 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:45,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:46,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:46,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:47,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:39:47,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:39:47,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:48,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:48,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:49,279 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:49,452 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:50,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:50,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,394 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:50,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:51,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:52,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:52,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:53,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:54,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:54,395 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:55,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:39:55,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:56,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:56,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:57,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:58,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:58,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:39:59,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:00,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:00,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:00,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,396 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:00,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:01,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:02,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:40:02,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:40:02,280 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:02,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:03,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:04,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:04,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:05,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:05,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:06,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:06,397 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:06,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:07,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:08,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:08,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:09,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:10,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,398 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:10,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:11,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:11,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:12,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:12,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:13,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:14,177 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:40:14,281 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:14,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:15,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:16,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:16,399 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:16,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:17,178 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:17,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:40:17,197 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:40:17,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:18,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:18,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:19,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:20,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:20,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,400 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:20,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:21,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:22,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:22,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:22,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:23,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:24,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:24,401 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:25,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:26,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:26,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:27,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:27,282 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:28,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:28,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:29,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:30,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:30,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,402 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:30,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:31,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:32,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:40:32,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:40:32,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:32,466 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:33,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:34,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:34,403 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:35,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:36,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:36,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:37,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:38,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:38,283 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:38,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:39,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:40,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:40,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,404 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:40,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:41,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:42,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:42,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:43,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:43,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:44,178 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:40:44,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:44,405 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:45,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:46,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:46,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:47,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:40:47,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:40:47,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:48,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:48,432 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:49,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:50,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:50,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,406 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:50,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:51,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:52,284 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:52,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:53,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:54,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:54,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:54,407 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:55,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:56,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:56,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:57,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:58,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:58,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:40:59,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:40:59,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:00,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:00,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,408 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:00,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:01,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:02,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:41:02,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:41:02,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:02,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:03,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:04,285 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:04,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:04,409 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:05,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:06,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:06,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:07,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:08,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:08,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:09,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:10,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:10,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:10,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,410 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:10,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:11,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:12,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:12,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:13,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:14,178 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:41:14,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:14,411 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:15,286 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:16,179 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:16,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:16,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:17,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:41:17,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:41:17,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:18,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:18,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:19,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:20,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:20,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,412 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:20,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:21,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:21,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:22,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:22,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:23,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:24,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:24,413 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:25,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:26,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:26,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:27,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:27,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:28,287 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:28,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:29,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:30,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:30,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,414 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:30,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:31,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:32,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:41:32,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:41:32,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:32,455 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:33,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:34,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:34,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:35,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:36,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:36,415 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:36,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:37,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:38,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:38,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:38,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:39,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:40,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,416 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:40,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:41,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:42,288 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:42,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:43,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:43,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:44,179 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:41:44,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:44,417 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:45,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:46,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:46,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:47,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:41:47,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:41:47,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:48,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:48,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:48,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:49,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:50,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:50,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,418 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:50,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:51,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:52,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:52,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:53,289 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:54,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:54,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:54,419 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:55,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:56,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:56,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:57,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:58,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:58,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:41:59,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:41:59,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:00,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:00,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,420 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:00,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:01,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:02,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:42:02,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:42:02,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:02,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:03,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:04,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:04,450 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:05,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:06,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:06,421 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:06,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:07,290 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:08,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:08,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:09,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:10,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:10,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,422 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:10,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:11,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:12,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:12,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:13,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:14,179 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:42:14,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:14,423 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:15,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:16,180 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:16,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:16,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:17,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:42:17,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:42:17,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:18,291 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:18,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:19,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:20,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:20,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,424 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:20,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:21,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:21,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:22,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:22,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:23,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:24,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:24,425 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:25,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:26,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:26,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:27,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:27,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:28,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:28,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:29,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:30,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:30,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,426 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:30,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:31,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:32,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:42:32,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:42:32,292 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:32,474 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:33,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:34,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:34,427 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:35,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:36,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:36,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:37,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:38,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:38,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:38,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:39,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:40,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:40,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,428 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:40,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:41,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:42,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:42,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:43,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:43,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:44,180 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:42:44,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:44,429 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:45,293 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:46,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:46,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:47,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:42:47,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:42:47,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:48,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:48,462 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:49,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:50,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:50,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,430 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:50,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:51,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:52,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:52,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:53,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:54,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:54,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:54,431 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:55,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:56,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:56,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:57,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:58,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:58,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:42:59,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:42:59,294 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:00,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:00,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,432 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:00,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:01,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:02,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:43:02,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:43:02,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:02,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:03,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:04,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:04,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:04,433 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:05,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:06,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:06,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:07,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:08,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:08,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:09,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:10,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:10,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:10,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,434 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:10,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:11,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:12,295 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:12,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:13,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:14,180 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:43:14,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:14,435 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:15,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:16,181 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:16,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:16,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:17,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:43:17,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:43:17,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:18,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:18,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:19,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:20,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:20,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,436 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:20,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:21,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:21,457 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:22,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:22,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:23,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:24,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:24,437 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:25,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:26,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:26,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:27,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:27,296 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:28,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:28,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:29,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:30,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:30,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,438 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:30,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:31,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:32,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:43:32,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:43:32,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:32,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:32,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:33,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:34,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:34,439 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:35,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:36,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:36,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:37,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:38,213 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:38,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:38,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:39,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:40,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:40,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,440 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:40,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:41,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:42,297 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:42,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:43,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:43,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:44,181 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:43:44,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:44,441 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:45,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:46,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:46,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:47,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:43:47,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:43:47,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:48,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:48,445 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:49,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:50,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:50,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,442 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:50,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:51,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:52,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:52,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:53,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:54,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:54,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:54,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:55,298 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:56,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:56,443 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:56,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:57,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:58,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:58,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:43:59,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:43:59,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:00,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:00,444 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:01,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:02,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:44:02,198 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:44:02,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:02,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:03,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:04,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:04,440 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:04,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:05,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:06,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:06,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,445 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:06,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:07,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:08,299 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:08,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:09,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:10,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:10,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:10,446 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:11,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:12,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:12,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:13,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:14,181 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:44:14,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:14,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:15,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:16,182 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:16,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:16,447 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:16,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:17,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:44:17,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:44:17,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:18,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:18,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:19,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:20,300 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,448 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:20,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:21,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:21,458 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:22,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:22,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:23,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:24,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:24,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:25,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:26,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:26,449 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:26,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:27,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:27,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:28,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:28,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:29,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:30,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,450 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:30,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:31,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:32,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:44:32,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:44:32,301 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:32,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:32,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:33,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:34,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:34,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:35,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:36,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:36,451 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:36,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:37,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:38,214 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:38,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:38,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:39,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:40,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,452 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:40,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:41,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:42,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:42,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:43,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:43,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:44,181 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:44:44,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:44,453 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:45,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:46,302 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:46,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:47,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:44:47,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:44:47,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:48,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:48,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:48,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:49,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:50,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:50,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,454 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:50,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:51,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:52,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:52,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:53,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:54,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:54,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:54,455 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:55,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:56,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:56,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:57,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:58,303 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:58,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:44:59,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:44:59,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:00,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:00,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,456 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:00,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:01,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:02,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:45:02,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:45:02,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:02,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:03,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:04,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:04,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:04,457 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:05,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:06,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:06,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:07,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:08,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:08,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:09,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:10,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:10,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:10,458 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:10,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:11,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:12,304 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:12,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:13,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:14,182 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:45:14,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,459 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:14,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:15,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:16,182 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:16,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:16,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:17,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:45:17,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:45:17,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:18,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:18,460 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:19,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:20,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:20,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:21,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:21,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:22,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:22,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:23,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:24,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:24,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,461 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:24,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:25,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:26,305 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:26,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:27,215 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:27,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:28,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:28,462 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:29,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:30,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:30,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:31,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:32,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:45:32,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:45:32,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:32,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:32,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:33,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:34,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:34,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,463 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:34,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:35,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:36,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:36,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:37,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:38,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:38,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:38,464 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:39,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:40,306 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:40,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:41,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:42,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:42,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:43,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:44,182 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:45:44,184 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:44,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:44,465 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:45:44,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,465 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:44,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:45,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:46,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:46,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:47,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:45:47,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:45:47,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:48,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:48,466 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:49,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:49,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:50,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:50,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:51,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:52,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:52,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:53,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:54,307 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,467 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:54,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:55,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:45:55,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:56,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:56,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:57,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:58,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:58,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:45:59,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:00,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:00,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:00,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,468 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:00,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:01,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:02,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:46:02,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:46:02,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:02,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:03,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:04,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:04,469 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:05,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:05,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:06,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:06,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:07,308 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:08,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:08,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:09,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:10,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:10,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,470 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:10,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:11,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:11,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:12,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:12,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:13,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:14,183 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:46:14,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:14,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:15,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:16,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:16,471 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:16,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:17,183 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:17,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:46:17,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:46:17,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:18,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:18,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:19,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:20,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:20,472 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:21,309 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:22,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:22,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:22,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:23,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:24,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:24,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:25,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:26,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:26,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,473 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:26,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:27,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:27,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:28,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:28,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:29,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:30,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:30,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:31,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:32,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:46:32,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:46:32,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:32,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:32,474 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:32,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:33,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:34,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:34,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:35,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:36,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:36,475 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:37,310 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:38,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:38,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:38,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:39,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:40,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:40,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:41,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:42,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:42,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,476 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:42,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:43,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:43,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:44,183 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:46:44,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:44,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:45,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:46,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:46,477 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:47,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:46:47,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:46:47,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:48,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:48,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:48,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:49,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:50,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:50,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:51,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:52,311 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:52,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,478 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:52,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:53,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:54,216 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:54,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:54,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:55,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:56,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:56,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:57,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:58,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:46:58,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,479 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:58,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:46:59,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:46:59,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:00,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:00,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:01,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:02,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:47:02,199 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:47:02,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:02,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:03,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:04,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:04,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:04,480 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:04,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:05,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:06,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:06,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:07,312 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:08,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:08,481 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:09,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:10,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:10,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:10,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:11,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:12,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:12,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:13,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:14,184 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:47:14,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:14,482 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:47:14,482 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:14,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:15,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:16,184 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:16,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:16,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:17,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:47:17,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:47:17,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:18,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:18,483 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:19,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:20,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:20,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:21,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:21,469 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:22,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:22,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:23,313 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:24,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:24,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,484 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:24,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:25,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:26,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:26,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:27,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:27,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:28,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:28,485 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:29,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:30,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:30,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:31,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:32,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:47:32,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:47:32,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:32,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:32,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:33,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:34,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:34,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,486 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:34,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:35,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:36,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:36,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:37,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:38,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:38,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:38,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:39,314 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:40,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:40,487 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:40,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:41,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:42,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:42,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:43,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:43,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:44,184 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:47:44,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,488 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:44,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:45,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:46,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:46,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:47,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:47:47,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:47:47,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:48,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:48,468 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:48,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:49,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:50,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:50,489 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:50,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:51,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:52,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:52,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:53,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:54,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:54,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:54,490 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:55,315 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:56,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:56,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:57,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:58,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:58,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:47:59,217 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:47:59,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:00,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:00,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,491 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:00,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:01,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:02,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:48:02,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:48:02,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:02,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:03,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:04,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:04,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:04,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:05,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:06,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:06,492 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:06,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:07,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:08,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:08,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:09,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:10,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:10,316 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:10,493 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:11,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:12,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:12,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:13,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:14,185 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:48:14,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:14,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:15,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:16,185 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:16,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:16,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,494 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:16,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:17,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:48:17,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:48:17,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:18,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:18,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:19,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:20,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:20,495 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:21,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:21,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:22,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:22,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:23,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:24,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:24,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:25,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:26,317 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:26,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,496 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:26,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:27,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:27,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:28,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:28,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:29,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:30,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:30,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:31,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:32,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:48:32,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:48:32,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:32,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:32,497 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:32,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:33,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:34,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:34,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:35,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:36,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:36,498 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:37,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:38,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:38,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:38,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:39,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:40,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:40,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:41,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:42,318 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:42,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,499 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:42,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:43,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:43,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:44,185 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:48:44,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:44,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:45,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:46,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:46,500 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:47,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:48:47,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:48:47,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:48,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:48,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:48,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:49,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:50,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:50,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:51,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:52,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:52,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,501 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:52,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:53,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:54,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:54,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:54,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:55,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:56,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:56,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:57,319 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:58,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:48:58,502 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:58,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:48:59,218 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:48:59,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:00,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:00,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:01,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:02,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:49:02,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:49:02,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:02,503 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:03,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:04,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:04,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:04,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:05,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:06,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:06,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:07,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:08,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:08,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,504 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:08,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:09,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:10,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:10,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:10,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:11,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:12,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:12,505 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:13,320 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:14,186 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:49:14,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:14,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:15,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:16,186 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:16,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:16,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:17,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:49:17,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:49:17,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:18,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:18,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,506 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:18,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:19,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:20,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:20,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:21,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:21,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:22,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:22,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:23,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:24,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:24,507 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:24,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:25,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:26,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:26,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:27,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:27,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:28,321 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:28,508 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:29,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:30,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:30,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:31,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:32,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:49:32,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:49:32,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:32,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:32,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:33,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:34,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:34,509 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:35,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:36,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:36,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:37,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:38,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:38,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:38,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:39,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:40,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:40,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,510 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:40,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:41,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:42,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:42,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:43,322 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:44,186 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:49:44,187 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:44,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:44,511 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:45,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:46,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:46,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:47,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:49:47,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:49:47,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:48,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:48,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:49,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:49,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:50,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,512 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:50,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:51,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:52,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:52,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:53,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:54,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:54,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:55,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:49:55,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:56,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:56,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,513 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:56,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:57,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:58,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:58,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:49:59,323 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:00,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:00,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:01,219 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:01,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:02,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:50:02,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:50:02,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:02,514 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:02,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:03,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:04,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:04,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:05,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:06,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:06,460 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:06,515 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:07,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:08,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:08,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:09,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:10,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:10,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:11,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:12,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:12,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:12,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,516 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:12,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:13,324 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:14,187 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:50:14,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:14,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:15,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:16,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:16,517 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:17,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:50:17,200 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:50:17,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:17,466 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:18,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:18,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:19,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:20,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:20,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:21,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:22,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,518 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:22,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:23,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:23,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:24,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:24,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:25,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:26,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:26,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:27,325 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:28,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:28,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:28,519 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:28,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:29,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:30,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:30,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:31,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:32,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:50:32,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:50:32,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:32,520 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:33,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:33,432 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:34,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:34,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:35,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:36,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:36,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:37,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:38,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:38,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,521 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:38,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:39,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:39,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:40,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:40,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:41,326 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:42,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:42,522 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:43,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:44,187 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:50:44,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:44,522 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:44,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:45,187 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:45,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:46,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:46,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:47,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:50:47,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:50:47,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:48,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:48,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,523 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:48,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:49,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:50,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:50,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:50,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:51,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:52,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:52,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:53,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:54,327 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:54,524 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:54,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:55,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:56,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:50:56,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:56,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:57,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:58,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:58,525 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:50:59,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:00,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:00,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:01,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:01,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:02,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:51:02,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:51:02,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:02,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:03,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:04,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:04,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,526 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:04,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:05,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:06,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:06,442 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:06,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:07,328 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:08,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:08,527 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:09,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:10,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:10,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:11,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:12,220 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:12,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:12,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:13,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:14,187 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:51:14,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:14,528 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:51:14,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,528 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:14,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:15,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:16,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:16,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:17,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:51:17,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:51:17,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:17,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:18,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:18,529 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:19,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:20,329 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:20,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:21,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:22,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:22,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:23,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:23,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:24,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:24,530 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:25,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:26,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:26,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:27,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:28,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:28,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:29,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:29,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:30,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:30,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,531 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:30,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:31,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:32,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:51:32,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:51:32,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:32,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:33,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:34,330 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:34,471 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:34,532 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:35,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:36,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:36,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:37,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:38,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:38,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:39,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:40,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:40,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:40,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,533 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:40,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:41,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:42,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:42,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:43,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:44,188 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:51:44,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:44,534 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:45,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:46,188 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:46,331 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:46,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:47,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:51:47,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:51:47,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:48,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:48,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:49,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:50,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:50,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,535 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:50,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:51,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:51,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:52,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:52,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:53,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:54,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:54,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:55,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:56,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:56,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,536 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:56,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:57,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:51:57,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:58,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:58,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:51:59,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:00,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:00,537 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:01,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:02,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:52:02,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:52:02,332 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:02,462 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:02,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:03,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:04,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:04,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:05,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:06,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:06,538 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:07,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:08,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:08,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:08,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:09,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:10,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:10,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:11,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:12,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,539 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:12,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:13,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:13,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:14,188 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:52:14,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:14,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:15,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:16,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:16,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:17,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:52:17,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:52:17,333 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:18,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:18,467 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:18,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,540 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:18,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:19,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:20,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:20,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:21,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:22,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:22,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:23,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:24,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:24,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:24,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,541 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:24,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:25,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:26,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:26,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:27,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:28,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:28,542 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:29,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:30,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:30,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:30,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:31,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:32,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:52:32,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:52:32,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:32,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:33,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:34,334 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:34,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,543 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:34,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:35,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:35,471 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:36,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:36,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:37,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:38,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:38,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:39,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:40,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:40,544 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:40,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:41,221 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:41,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:42,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:42,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:43,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:44,191 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:52:44,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:44,545 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:45,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:46,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:46,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:47,191 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:47,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:52:47,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:52:47,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:48,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:48,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:49,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:50,335 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:50,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,546 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:50,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:51,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:52,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:52,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:52,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:53,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:54,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:54,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:55,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:56,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:56,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,547 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:56,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:57,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:58,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:52:58,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:58,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:52:59,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:00,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:00,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:01,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:02,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:53:02,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:53:02,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:02,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,548 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:02,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:03,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:03,481 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:04,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:04,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:05,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:06,336 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:06,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:07,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:08,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:08,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,549 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:08,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:09,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:09,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:10,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:10,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:11,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:12,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:12,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:13,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:14,189 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:53:14,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:14,550 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:53:14,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,550 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:14,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:15,190 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:15,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:16,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:16,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:17,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:53:17,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:53:17,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:18,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:18,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:19,337 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:20,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:20,443 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:20,551 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:20,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:21,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:22,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:22,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:23,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:24,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:24,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:25,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:26,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:26,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:26,552 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:26,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:27,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:28,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:28,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:29,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:30,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:30,553 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:31,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:31,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:32,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:53:32,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:53:32,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:32,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:33,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:34,338 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:34,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:35,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:36,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:36,466 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:36,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,554 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:36,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:37,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:38,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:38,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:39,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:40,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:40,555 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:41,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:42,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:42,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:42,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:43,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:44,191 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:53:44,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:44,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:45,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:46,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:46,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,556 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:46,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:47,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:53:47,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:53:47,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:47,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:48,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:48,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:49,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:50,339 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:50,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:51,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:52,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:52,557 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:52,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:53,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:53,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:54,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:54,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:55,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:56,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:56,558 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:57,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:58,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:58,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:53:59,222 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:53:59,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:00,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:00,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:01,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:02,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:54:02,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:54:02,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:02,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,559 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:02,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:03,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:04,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:04,469 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:04,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:05,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:06,340 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:06,560 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:07,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:08,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:08,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:09,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:10,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:10,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:10,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:11,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:12,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:12,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,561 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:12,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:13,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:14,190 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:54:14,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:14,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:15,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:16,191 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:16,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:16,562 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:17,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:54:17,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:54:17,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:18,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:18,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:19,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:20,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:20,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:21,341 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:21,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:22,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,563 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:22,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:23,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:24,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:24,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:25,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:26,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:26,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:27,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:27,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:28,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:28,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,564 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:28,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:29,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:30,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:30,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:31,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:32,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:54:32,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:54:32,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:32,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:32,565 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:33,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:34,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:34,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:35,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:36,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:36,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:37,342 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:38,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:38,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:38,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,566 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:38,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:39,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:40,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:40,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:41,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:42,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:42,567 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:43,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:43,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:44,191 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:54:44,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:44,567 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:44,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:45,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:46,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:46,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:47,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:54:47,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:54:47,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:48,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:48,441 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,568 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:48,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:49,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:50,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:50,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:51,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:52,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:52,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:53,343 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:54,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:54:54,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:54,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,569 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:54,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:55,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:56,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:56,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:57,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:58,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:58,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:54:59,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:00,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:00,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:00,570 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:00,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:01,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:02,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:55:02,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:55:02,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:02,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:03,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:04,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:04,571 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:05,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:05,477 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:06,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:06,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:07,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:08,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:08,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:09,344 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:10,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,572 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:10,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:11,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:11,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:12,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:12,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:13,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:14,191 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:55:14,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:14,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:15,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:16,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:16,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,573 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:16,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:17,192 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:17,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:55:17,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:55:17,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:18,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:18,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:19,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:20,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:20,574 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:21,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:22,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:22,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:22,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:23,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:24,345 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:24,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:25,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:26,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:26,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,575 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:26,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:27,223 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:27,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:28,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:28,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:29,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:30,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:30,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:31,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:32,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:55:32,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:55:32,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:32,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:32,576 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:32,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:33,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:34,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:34,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:35,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:36,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:36,577 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:37,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:38,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:38,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:38,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:39,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:40,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:40,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:41,346 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:42,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,578 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:42,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:43,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:43,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:44,192 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:55:44,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:44,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:45,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:46,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:46,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:47,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:55:47,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:55:47,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:48,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:48,464 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:48,579 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:48,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:49,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:50,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:50,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:51,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:52,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:52,580 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:53,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:54,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:54,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:54,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:55,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:56,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:56,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:57,347 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:58,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,581 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:58,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:55:59,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:55:59,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:00,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:00,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:01,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:02,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:56:02,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:56:02,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:02,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:03,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:04,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:04,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:04,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,582 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:04,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:05,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:06,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:06,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:07,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:08,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:08,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:09,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:10,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:10,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:10,583 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:10,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:11,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:12,348 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:12,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:13,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:14,192 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:56:14,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:14,584 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:15,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:16,193 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:16,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:16,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:17,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:56:17,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:56:17,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:18,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:18,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:19,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:20,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:20,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,585 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:20,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:21,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:21,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:22,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:22,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:23,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:24,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:24,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:25,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:26,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:26,586 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:26,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:27,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:27,349 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:28,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:28,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:29,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:30,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:30,587 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:31,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:32,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:56:32,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:56:32,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:32,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:32,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:33,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:34,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:34,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:35,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:36,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,588 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:36,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:37,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:38,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:38,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:38,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:39,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:40,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:40,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:41,350 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:42,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:42,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,589 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:42,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:43,224 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:43,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:44,193 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:56:44,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:44,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:45,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:46,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:46,590 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:47,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:56:47,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:56:47,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:48,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:48,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:48,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:49,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:50,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:50,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:51,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:52,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,591 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:52,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:53,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:54,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:56:54,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:54,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:55,351 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:56,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:56,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:57,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:58,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:56:58,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,592 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:58,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:56:59,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:00,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:00,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:00,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:01,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:02,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:57:02,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:57:02,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:02,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:03,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:04,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:04,593 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:04,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:05,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:05,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:06,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:06,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:07,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:08,352 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:08,594 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:09,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:10,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:10,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:11,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:11,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:12,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:12,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:13,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:14,193 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:57:14,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:14,595 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:57:14,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,595 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:14,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:15,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:16,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:16,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:17,194 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:17,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:57:17,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:57:17,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:18,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:18,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:19,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:20,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:20,596 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:20,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:21,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:22,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:22,353 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:22,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:23,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:24,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:24,597 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:25,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:26,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:26,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:27,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:27,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:28,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:28,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:29,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:30,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:30,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,598 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:30,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:31,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:32,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:57:32,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:57:32,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:32,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:32,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:33,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:34,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:34,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:35,354 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:36,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:36,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,599 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:36,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:37,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:38,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:38,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:38,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:39,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:40,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:40,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:41,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:42,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:42,600 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:42,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:43,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:43,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:44,194 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:57:44,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:44,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:45,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:46,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:46,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,601 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:46,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:47,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:57:47,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:57:47,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:48,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:48,467 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:48,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:49,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:50,355 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:50,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:51,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:52,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:52,602 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:52,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:53,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:54,225 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:54,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:54,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:55,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:56,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:56,603 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:57,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:58,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:58,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:57:59,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:57:59,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:00,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:00,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:01,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:02,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:58:02,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:58:02,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,604 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:02,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:03,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:04,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:04,437 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:04,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:05,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:06,356 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:06,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:07,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:08,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:08,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,605 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:08,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:09,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:10,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:10,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:10,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:11,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:12,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:12,606 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:13,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:14,194 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:58:14,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:14,606 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:14,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:15,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:16,195 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:16,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:16,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:17,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:58:17,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:58:17,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:18,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,607 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:18,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:19,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:20,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:20,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:21,357 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:21,441 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:22,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:22,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:23,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:24,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:24,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,608 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:24,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:25,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:26,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:26,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:27,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:27,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:28,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:28,609 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:29,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:30,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:30,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:31,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:32,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:58:32,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:58:32,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:32,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:32,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:33,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:34,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,610 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:34,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:35,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:36,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:36,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:37,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:38,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:38,358 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:38,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:39,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:40,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:40,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,611 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:40,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:41,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:42,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:42,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:43,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:44,195 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:58:44,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:44,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:44,612 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:45,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:46,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:46,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:47,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:58:47,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:58:47,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:48,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:48,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:49,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:49,458 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:50,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:50,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,613 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:50,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:51,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:52,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:52,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:53,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:54,359 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:54,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:55,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:58:55,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:56,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:56,614 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:56,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:57,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:58,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:58,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:58:59,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:00,226 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:00,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:00,615 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:01,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:02,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:59:02,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:59:02,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:02,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:03,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:04,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:04,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:05,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:05,461 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:06,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:06,616 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:07,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:08,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:08,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:09,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:10,360 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:10,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:11,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:11,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:12,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:12,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,617 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:12,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:13,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:14,195 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:59:14,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:14,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:15,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:16,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:16,618 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:17,196 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:17,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:59:17,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:59:17,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:18,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:18,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:19,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:20,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:20,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:21,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:22,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:22,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,619 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:22,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:23,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:24,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:24,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:25,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:26,361 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:26,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:27,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:27,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:28,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:28,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,620 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:28,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:29,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:30,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:30,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:31,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:32,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:59:32,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:59:32,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:32,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:32,621 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:33,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:34,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:34,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:35,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:36,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:36,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:37,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:38,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:38,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,622 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:38,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:39,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:40,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:40,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:41,362 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:42,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:42,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:43,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:44,195 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-03 23:59:44,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:44,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:44,623 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-03 23:59:44,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,623 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:44,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:45,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:46,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:46,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:47,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-03 23:59:47,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-03 23:59:47,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:48,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:48,624 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:49,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:49,435 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:50,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:50,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:51,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:52,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:52,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:53,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:54,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:54,625 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:55,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-03 23:59:55,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:56,363 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:56,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:57,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:58,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:58,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-03 23:59:59,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:00,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:00,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,626 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:00,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:01,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:01,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:02,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:00:02,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:00:02,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:02,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:03,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:04,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:04,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:05,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:06,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:06,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:06,627 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:06,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:07,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:08,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:08,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:09,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:10,364 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,628 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:10,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:11,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:12,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:12,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:12,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:13,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:14,196 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:00:14,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:14,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:15,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:16,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:16,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,629 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:16,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:17,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:00:17,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:00:17,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:17,441 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:18,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:18,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:19,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:20,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:20,630 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:21,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:22,365 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:22,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:23,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:23,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:24,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:24,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:25,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:26,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:26,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,631 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:26,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:27,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:28,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:28,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:29,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:29,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:30,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:30,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:31,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:32,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:00:32,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:00:32,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:32,632 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:32,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:33,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:34,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:34,447 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:34,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:35,366 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:36,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:36,633 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:37,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:38,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:38,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:39,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:40,227 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:40,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:40,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:41,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:42,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:42,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,634 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:42,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:43,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:44,196 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:00:44,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:44,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:45,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:46,197 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:46,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:46,635 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:47,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:00:47,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:00:47,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:48,367 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:48,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:49,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:50,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:50,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:51,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:51,458 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:52,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:52,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,636 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:52,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:53,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:54,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:54,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:55,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:56,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:56,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:57,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:00:57,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:58,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:00:58,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,637 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:58,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:00:59,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:00,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:00,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:01,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:02,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:01:02,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:01:02,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:02,445 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:02,638 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:03,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:04,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:04,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:05,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:06,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:06,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:07,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:08,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:08,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:08,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,639 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:08,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:09,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:10,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:10,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:11,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:12,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:12,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:13,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:13,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:14,197 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:01:14,369 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:14,640 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:01:14,640 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:14,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:15,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:16,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:16,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:17,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:01:17,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:01:17,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:18,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:18,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:18,641 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:19,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:20,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:20,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:21,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:22,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:22,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:23,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:24,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:24,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,642 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:24,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:25,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:26,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:26,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:27,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:28,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:28,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:29,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:30,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:30,370 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:30,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,643 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:30,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:31,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:32,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:01:32,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:01:32,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:32,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:33,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:34,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:34,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:35,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:35,460 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:36,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:36,644 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:36,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:37,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:38,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:38,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:39,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:40,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:40,645 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:41,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:41,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:42,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:42,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:43,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:44,197 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:01:44,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:44,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:45,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:46,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:46,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,646 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:46,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:47,198 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:47,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:01:47,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:01:47,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:48,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:48,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:49,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:50,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:50,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:51,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:52,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:52,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:52,647 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:52,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:53,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:54,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:54,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:55,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:56,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:56,648 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:57,228 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:01:57,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:58,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:58,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:01:59,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:00,372 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:00,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:01,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:02,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:02:02,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:02:02,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:02,472 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:02,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,649 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:02,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:03,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:04,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:04,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:05,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:06,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:06,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:07,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:08,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:08,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:08,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,650 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:08,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:09,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:10,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:10,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:11,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:12,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:12,651 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:13,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:14,198 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:02:14,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:14,373 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:14,651 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:14,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:15,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:16,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:16,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:17,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:02:17,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:02:17,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:18,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:18,652 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:19,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:19,466 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:20,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:20,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:21,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:22,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:22,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:23,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:24,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:24,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,653 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:24,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:25,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:25,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:26,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:26,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:27,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:28,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:28,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:29,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:30,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:30,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,654 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:30,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:31,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:31,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:32,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:02:32,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:02:32,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:32,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:33,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:34,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:34,655 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:35,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:36,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:36,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:36,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:37,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:38,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:38,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:39,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:40,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:40,656 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:41,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:42,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:42,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:42,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:43,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:44,198 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:02:44,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:44,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:45,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:46,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:46,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,657 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:46,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:47,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:02:47,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:02:47,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:47,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:48,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:48,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:49,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:50,375 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:50,658 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:51,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:52,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:52,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:53,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:53,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:54,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:54,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:55,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:56,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,659 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:56,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:57,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:58,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:58,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:02:59,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:02:59,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:00,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:00,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:01,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:02,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:03:02,204 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:03:02,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:02,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,660 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:02,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:03,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:04,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:04,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:04,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:05,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:06,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:06,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:07,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:08,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:08,661 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:08,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:09,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:10,229 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:10,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:10,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:11,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:12,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:12,662 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:13,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:14,199 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:03:14,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:14,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:15,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:16,199 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:16,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:16,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:17,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:03:17,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:03:17,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:18,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:18,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,663 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:18,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:19,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:20,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:20,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:21,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:21,453 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:22,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:22,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:23,377 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:24,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:24,664 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:24,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:25,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:26,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:26,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:27,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:27,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:28,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:28,665 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:29,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:30,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:30,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:31,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:32,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:03:32,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:03:32,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:32,463 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:32,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:33,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:34,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:34,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,666 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:34,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:35,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:36,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:36,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:37,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:38,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:38,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:38,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:39,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:40,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:40,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,667 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:40,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:41,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:42,378 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:42,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:43,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:43,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:44,199 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:03:44,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:44,668 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:45,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:46,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:46,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:47,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:03:47,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:03:47,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:48,368 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:48,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:48,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:49,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:50,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,669 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:50,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:51,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:52,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:52,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:53,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:54,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:54,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:54,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:55,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:56,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:56,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,670 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:56,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:57,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:58,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:58,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:03:59,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:03:59,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:00,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:00,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:01,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:02,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:04:02,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:04:02,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:02,671 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:02,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:03,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:04,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:04,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:04,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:05,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:06,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:06,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:07,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:08,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:08,672 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:08,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:09,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:10,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:10,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:10,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:11,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:12,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:12,673 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:13,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:14,200 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:04:14,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:14,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:15,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:16,200 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:16,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:16,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:17,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:04:17,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:04:17,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:18,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:18,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,674 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:18,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:19,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:20,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:20,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:21,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:21,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:22,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:22,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:23,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:24,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:24,675 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:24,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:25,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:26,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:26,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:27,230 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:27,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:28,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:28,676 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:29,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:30,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:30,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:31,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:32,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:04:32,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:04:32,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:32,453 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:32,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:33,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:34,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:34,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,677 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:34,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:35,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:36,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:36,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:37,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:38,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:38,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:38,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:39,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:40,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:40,678 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:40,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:41,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:42,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:42,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:43,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:43,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:44,200 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:04:44,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,679 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:44,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:45,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:46,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:46,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:47,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:04:47,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:04:47,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:48,379 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:48,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:48,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:49,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:50,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:50,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,680 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:50,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:51,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:52,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:52,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:53,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:54,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:04:54,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:54,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:55,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:56,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:56,681 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:56,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:57,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:58,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:58,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:04:59,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:00,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:00,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:00,682 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:01,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:02,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:05:02,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:05:02,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:02,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:03,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:04,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:04,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:05,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:05,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:06,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:06,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,683 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:06,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:07,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:08,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:08,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:09,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:10,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:10,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:11,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:11,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:12,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:12,684 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:12,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:13,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:14,201 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:05:14,384 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:14,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:15,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:16,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:16,685 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:17,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:17,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:05:17,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:05:17,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:18,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:18,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:19,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:20,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:20,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:21,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:22,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:22,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:22,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,686 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:22,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:23,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:24,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:24,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:25,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:26,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:26,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:27,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:27,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:28,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:28,687 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:28,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:29,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:30,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:30,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:31,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:32,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:05:32,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:05:32,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:32,471 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:32,688 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:33,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:34,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:34,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:35,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:36,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:36,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:37,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:38,231 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:38,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,689 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:38,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:39,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:40,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:40,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:41,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:42,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:42,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:43,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:43,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:44,201 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:05:44,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:44,690 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:05:44,690 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:44,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:45,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:46,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:46,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:47,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:05:47,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:05:47,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:48,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:48,468 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:48,691 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:49,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:50,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:50,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:51,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:52,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:52,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:53,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:54,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:54,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,692 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:54,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:55,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:56,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:56,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:57,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:58,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:58,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:05:59,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:05:59,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:00,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:00,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,693 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:00,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:01,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:02,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:06:02,205 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:06:02,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:02,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:03,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:04,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:04,448 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:04,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:05,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:06,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:06,694 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:06,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:07,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:08,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:08,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:09,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:10,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:10,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:10,695 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:11,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:12,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:12,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:13,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:14,201 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:06:14,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:14,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:15,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:16,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:16,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:16,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,696 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:16,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:17,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:06:17,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:06:17,388 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:18,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:18,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:19,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:20,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:20,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:21,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:21,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:22,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:22,697 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:22,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:23,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:24,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:24,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:25,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:26,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:26,698 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:27,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:27,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:28,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:28,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:29,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:30,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:30,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:31,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:32,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:06:32,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:06:32,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:32,461 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:32,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,699 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:32,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:33,389 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:34,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:34,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:35,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:36,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:36,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:37,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:38,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:38,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:38,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,700 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:38,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:39,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:40,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:40,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:41,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:42,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:42,701 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:43,232 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:43,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:44,202 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:06:44,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:44,701 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:44,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:45,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:46,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:46,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:47,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:06:47,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:06:47,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:48,390 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:48,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:48,702 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:49,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:50,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:50,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:51,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:52,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:52,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:53,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:54,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:54,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:54,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,703 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:54,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:55,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:56,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:56,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:57,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:58,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:58,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:06:59,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:06:59,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:00,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:00,704 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:00,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:01,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:02,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:07:02,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:07:02,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:02,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:03,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:04,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:04,467 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:04,705 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:05,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:06,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:06,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:07,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:08,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:08,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:09,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:10,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:10,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:10,706 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:11,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:12,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:12,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:13,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:14,202 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:07:14,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:14,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:15,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:16,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:16,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:16,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,707 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:16,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:17,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:07:17,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:07:17,392 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:18,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:18,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:19,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:20,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:20,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:21,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:21,463 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:22,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:22,708 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:22,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:23,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:24,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:24,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:25,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:26,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:26,709 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:27,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:27,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:28,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:28,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:29,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:30,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:30,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:31,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:32,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:07:32,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:07:32,387 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:32,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:32,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,710 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:32,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:33,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:34,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:34,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:35,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:36,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:36,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:37,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:38,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:38,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:38,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,711 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:38,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:39,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:40,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:40,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:41,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:42,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:42,712 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:43,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:43,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:44,203 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:07:44,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:44,712 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:44,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:45,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:46,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:46,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:47,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:07:47,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:07:47,394 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:48,376 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:48,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,713 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:48,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:49,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:50,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:50,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:51,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:52,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:52,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:53,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:54,233 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:54,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:54,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,714 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:54,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:55,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:56,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:56,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:57,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:58,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:58,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:07:59,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:07:59,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:00,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:00,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,715 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:00,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:01,395 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:02,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:08:02,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:08:02,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:02,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:03,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:04,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:04,448 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:04,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:05,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:06,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:06,716 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:06,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:07,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:08,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:08,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:09,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:10,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:10,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:10,717 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:11,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:12,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:12,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:13,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:14,203 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:08:14,396 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:14,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:15,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:16,204 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:16,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:16,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,718 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:16,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:17,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:08:17,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:08:17,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:18,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:18,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:19,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:20,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:20,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:21,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:21,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:22,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:22,719 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:22,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:23,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:24,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:24,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:25,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:26,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:26,720 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:27,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:27,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:28,397 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:28,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:29,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:30,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:30,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:31,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:32,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:08:32,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:08:32,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:32,439 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:32,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,721 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:32,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:33,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:34,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:34,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:35,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:36,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:36,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:37,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:38,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:38,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:38,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,722 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:38,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:39,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:40,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:40,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:41,398 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:42,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:42,723 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:43,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:43,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:44,206 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:08:44,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:44,723 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:44,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:45,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:46,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:46,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:47,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:08:47,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:08:47,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:48,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:48,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:48,724 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:49,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:50,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:50,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:51,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:52,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:52,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:53,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:54,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:54,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:54,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,725 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:54,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:55,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:56,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:56,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:57,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:58,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:58,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:08:59,234 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:08:59,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:00,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:00,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,726 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:00,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:01,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:02,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:09:02,206 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:09:02,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:02,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:03,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:04,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:04,446 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:04,727 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:05,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:06,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:06,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:07,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:08,400 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:08,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:09,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:10,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:10,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:10,728 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:11,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:12,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:12,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:13,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:14,204 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:09:14,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:14,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:15,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:16,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:16,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:16,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,729 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:16,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:17,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:09:17,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:09:17,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:18,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:18,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:19,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:20,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:20,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:21,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:21,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:22,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:22,730 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:22,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:23,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:24,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:24,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:25,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:26,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:26,731 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:27,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:27,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:28,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:28,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:29,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:30,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:30,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:31,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:32,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:09:32,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:09:32,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:32,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:32,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,732 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:32,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:33,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:34,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:34,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:35,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:36,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:36,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:37,402 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:38,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:38,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:38,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,733 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:38,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:39,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:40,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:40,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:41,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:42,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:42,734 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:43,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:44,206 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:09:44,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:44,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:44,734 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:44,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:45,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:46,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:46,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:47,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:09:47,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:09:47,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:48,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:48,735 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:49,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:49,469 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:50,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:50,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:51,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:52,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:52,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:53,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:54,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:54,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,736 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:54,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:55,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:09:55,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:56,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:56,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:57,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:58,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:58,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:09:59,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:00,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:00,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:00,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,737 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:00,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:01,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:02,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:10:02,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:10:02,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:02,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:03,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:04,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:04,738 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:05,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:05,510 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:06,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:06,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:07,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:08,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:08,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:09,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:10,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:10,739 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:11,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:11,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:12,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:12,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:13,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:14,205 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:10:14,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:14,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:15,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:16,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:16,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,740 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:16,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:17,206 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:17,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:10:17,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:10:17,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:18,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:18,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:19,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:20,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:20,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:21,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:22,235 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:22,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:22,741 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:22,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:23,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:24,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:24,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:25,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:26,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:26,742 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:27,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:27,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:28,405 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:28,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:29,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:30,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:30,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:31,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:32,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:10:32,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:10:32,374 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:32,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:32,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,743 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:32,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:33,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:34,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:34,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:35,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:36,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:36,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:37,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:38,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:38,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:38,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,744 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:38,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:39,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:40,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:40,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:41,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:42,406 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:42,745 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:43,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:44,206 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:10:44,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:44,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:44,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:45,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:46,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:46,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:47,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:10:47,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:10:47,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:48,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:48,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,746 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:48,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:49,391 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:49,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:50,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:50,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:51,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:52,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:52,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:53,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:54,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:54,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,747 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:54,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:55,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:10:55,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:56,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:56,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:57,407 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:58,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:58,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:10:59,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:00,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:00,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:00,748 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:00,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:01,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:02,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:11:02,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:11:02,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:02,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:03,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:04,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:04,749 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:05,385 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:05,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:06,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:06,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:07,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:08,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:08,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:09,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:10,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:10,750 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:11,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:11,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:12,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:12,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:13,408 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:14,206 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:11:14,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:14,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:15,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:16,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:16,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,751 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:16,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:17,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:17,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:11:17,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:11:17,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:18,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:18,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:19,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:20,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:20,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:21,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:22,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:22,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:22,752 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:22,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:23,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:24,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:24,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:25,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:26,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:26,753 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:27,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:27,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:28,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:28,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:29,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:30,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:30,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:31,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:32,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:11:32,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:11:32,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:32,465 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:32,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,754 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:32,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:33,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:34,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:34,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:35,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:36,409 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:36,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:37,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:38,236 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:38,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:38,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,755 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:38,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:39,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:40,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:40,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:41,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:42,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:42,756 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:43,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:44,207 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:11:44,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:44,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:44,756 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:44,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:45,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:46,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:46,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:47,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:11:47,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:11:47,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:48,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:48,757 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:49,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:49,471 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:50,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:50,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:51,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:52,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:52,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:53,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:54,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:54,758 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:55,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:11:55,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:56,410 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:56,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:57,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:58,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:58,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:11:59,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:00,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:00,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:00,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,759 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:00,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:01,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:02,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:12:02,207 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:12:02,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:02,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:03,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:04,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:04,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:05,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:05,473 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:06,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:06,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,760 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:06,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:07,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:08,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:08,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:09,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:10,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:10,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:11,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:11,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:12,411 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:12,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,761 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:12,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:13,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:14,207 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:12:14,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:14,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:15,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:16,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:16,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:17,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:12:17,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:12:17,380 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:17,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:18,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:18,762 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:18,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:19,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:20,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:20,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:21,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:22,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:22,763 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:23,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:23,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:24,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:24,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:25,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:26,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:26,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:27,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:28,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:28,412 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,764 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:28,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:29,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:30,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:30,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:31,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:32,207 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:12:32,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:12:32,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:32,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:33,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:33,450 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:34,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:34,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,765 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:34,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:35,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:36,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:36,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:37,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:38,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:38,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:39,237 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:39,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:40,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:40,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,766 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:40,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:41,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:42,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:42,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:43,413 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:44,208 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:12:44,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:44,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:45,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:45,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:46,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:46,767 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:46,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:47,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:12:47,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:12:47,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:48,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:48,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:49,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:50,382 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:50,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:50,768 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:51,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:52,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:52,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:53,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:54,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:54,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:55,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:56,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:12:56,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:56,769 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:57,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:58,414 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:58,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:12:59,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:00,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:00,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:01,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:01,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:02,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:13:02,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:13:02,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:02,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,770 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:02,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:03,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:04,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:04,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:05,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:06,399 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:06,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:06,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:07,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:08,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:08,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,771 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:08,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:09,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:10,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:10,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:11,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:12,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:12,415 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:12,772 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:13,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:14,208 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:13:14,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:14,772 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:14,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:15,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:16,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:16,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:17,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:13:17,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:13:17,393 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:17,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:18,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:18,773 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:19,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:20,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:20,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:21,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:22,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:22,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:23,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:23,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:24,416 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:24,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,774 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:24,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:25,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:26,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:26,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:27,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:28,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:28,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:28,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:29,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:30,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:30,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,775 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:30,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:31,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:32,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:13:32,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:13:32,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:32,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:33,403 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:33,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:34,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:34,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:35,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:36,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:36,776 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:36,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:37,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:38,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:38,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:39,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:39,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:40,417 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:40,777 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:41,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:42,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:42,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:43,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:44,208 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:13:44,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:44,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:45,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:45,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:46,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:46,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,778 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:46,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:47,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:13:47,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:13:47,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:48,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:48,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:49,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:50,404 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:50,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:50,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:51,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:52,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:52,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,779 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:52,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:53,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:54,418 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:54,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:55,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:56,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:13:56,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:56,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:57,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:58,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:13:58,780 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:58,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:13:59,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:00,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:00,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:01,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:01,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:02,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:14:02,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:14:02,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:02,781 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:03,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:04,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:04,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:05,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:06,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:06,614 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:06,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:07,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:08,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:08,782 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:09,419 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:10,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:10,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:11,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:12,238 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:12,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:12,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:13,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:14,209 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:14:14,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:14,783 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:14:14,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,783 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:14,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:15,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:16,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:16,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:17,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:14:17,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:14:17,383 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:17,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:18,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:18,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:19,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:20,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:20,784 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:20,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:21,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:22,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:22,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:23,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:23,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:24,420 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:24,785 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:25,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:26,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:26,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:27,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:28,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:28,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:28,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:29,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:30,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,786 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:30,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:31,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:32,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:14:32,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:14:32,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:32,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:33,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:33,469 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:34,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:34,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:35,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:36,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:36,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,787 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:36,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:37,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:38,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:38,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:39,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:39,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:40,421 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:40,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:41,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:42,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:42,788 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:42,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:43,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:44,209 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:14:44,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:44,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:45,210 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:45,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:46,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,789 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:46,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:47,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:14:47,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:14:47,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:48,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:48,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:49,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:50,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:50,450 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:50,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:51,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:52,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:52,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,790 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:52,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:53,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:54,422 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:54,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:55,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:56,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:14:56,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:56,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:57,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:58,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:14:58,791 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:58,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:14:59,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:00,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:00,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:01,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:01,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:02,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:15:02,208 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:15:02,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:02,792 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:03,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:04,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:04,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:05,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:06,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:06,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:06,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:07,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:08,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:08,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,793 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:08,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:09,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:10,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:10,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:11,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:12,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:12,423 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:12,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:13,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:14,210 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:15:14,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:14,794 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:15:14,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,794 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:14,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:15,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:16,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:16,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:17,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:15:17,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:15:17,401 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:17,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:18,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:18,795 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:19,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:20,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:20,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:21,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:22,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:22,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:23,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:23,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:24,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:24,796 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:25,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:26,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:26,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:27,424 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:28,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:28,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:29,239 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:29,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:30,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:30,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,797 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:30,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:31,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:32,208 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:15:32,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:15:32,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:32,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:33,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:34,386 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:34,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:34,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:35,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:36,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:36,798 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:36,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:37,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:38,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:38,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:39,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:40,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:40,425 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:40,799 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:41,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:42,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:42,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:43,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:44,210 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:15:44,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:44,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:45,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:46,211 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:46,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,800 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:46,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:47,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:15:47,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:15:47,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:48,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:48,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:49,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:50,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:50,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:51,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:51,470 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:52,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:52,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,801 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:52,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:53,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:54,426 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:54,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:55,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:56,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:56,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:57,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:15:57,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:58,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:15:58,802 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:58,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:15:59,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:00,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:00,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:01,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:02,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:16:02,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:16:02,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:02,695 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:02,803 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:03,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:04,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:04,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:05,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:06,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:06,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:07,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:08,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:08,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,804 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:08,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:09,427 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:10,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:10,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:11,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:12,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:12,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:13,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:13,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:14,211 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:16:14,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:14,805 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:16:14,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,805 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:14,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:15,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:16,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:16,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:17,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:16:17,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:16:17,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:18,381 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:18,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:18,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:19,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:20,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:20,806 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:20,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:21,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:22,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:22,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:23,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:24,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:24,428 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:24,807 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:25,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:26,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:26,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:27,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:28,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:28,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:29,240 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:29,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:30,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,808 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:30,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:31,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:32,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:16:32,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:16:32,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:32,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:33,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:34,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:34,462 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:34,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:35,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:36,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:36,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,809 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:36,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:37,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:38,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:38,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:39,429 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:40,241 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:40,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:40,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:41,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:42,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:42,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,810 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:42,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:43,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:44,211 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:16:44,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:44,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:45,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:46,212 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:46,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:46,811 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:47,209 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: stop_status 2024-07-04 00:16:47,209 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: stop_status 2024-07-04 00:16:47,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:48,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:48,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:49,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:50,430 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:50,812 ERROR gpu :152382 [interfaces.py:monitor():142] Failed to sample metric: Unknown Error 2024-07-04 00:16:51,017 DEBUG SenderThread:152382 [sender.py:send():379] send: exit 2024-07-04 00:16:51,017 INFO SenderThread:152382 [sender.py:send_exit():586] handling exit code: 255 2024-07-04 00:16:51,018 INFO SenderThread:152382 [sender.py:send_exit():588] handling runtime: 3336 2024-07-04 00:16:51,023 INFO SenderThread:152382 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-04 00:16:51,024 INFO SenderThread:152382 [sender.py:send_exit():594] send defer 2024-07-04 00:16:51,024 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,024 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 0 2024-07-04 00:16:51,024 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,024 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 0 2024-07-04 00:16:51,024 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 1 2024-07-04 00:16:51,024 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,024 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 1 2024-07-04 00:16:51,024 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,024 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 1 2024-07-04 00:16:51,024 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 2 2024-07-04 00:16:51,024 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,024 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 2 2024-07-04 00:16:51,024 INFO HandlerThread:152382 [system_monitor.py:finish():203] Stopping system monitor 2024-07-04 00:16:51,025 ERROR gpu :152382 [interfaces.py:aggregate():159] Failed to serialize metric: division by zero 2024-07-04 00:16:51,025 INFO HandlerThread:152382 [interfaces.py:finish():200] Joined cpu monitor 2024-07-04 00:16:51,025 DEBUG SystemMonitor:152382 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-07-04 00:16:51,025 INFO HandlerThread:152382 [interfaces.py:finish():200] Joined disk monitor 2024-07-04 00:16:51,025 DEBUG SystemMonitor:152382 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-07-04 00:16:51,025 INFO HandlerThread:152382 [interfaces.py:finish():200] Joined gpu monitor 2024-07-04 00:16:51,026 INFO HandlerThread:152382 [interfaces.py:finish():200] Joined memory monitor 2024-07-04 00:16:51,026 INFO HandlerThread:152382 [interfaces.py:finish():200] Joined network monitor 2024-07-04 00:16:51,026 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,026 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 2 2024-07-04 00:16:51,026 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 3 2024-07-04 00:16:51,026 DEBUG SenderThread:152382 [sender.py:send():379] send: stats 2024-07-04 00:16:51,026 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,027 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 3 2024-07-04 00:16:51,027 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,027 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 3 2024-07-04 00:16:51,027 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 4 2024-07-04 00:16:51,027 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,027 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 4 2024-07-04 00:16:51,027 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,027 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 4 2024-07-04 00:16:51,027 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 5 2024-07-04 00:16:51,027 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,027 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 5 2024-07-04 00:16:51,027 DEBUG SenderThread:152382 [sender.py:send():379] send: summary 2024-07-04 00:16:51,030 INFO SenderThread:152382 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-07-04 00:16:51,031 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,031 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 5 2024-07-04 00:16:51,031 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 6 2024-07-04 00:16:51,031 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,031 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 6 2024-07-04 00:16:51,031 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,031 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 6 2024-07-04 00:16:51,031 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 7 2024-07-04 00:16:51,031 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: status_report 2024-07-04 00:16:51,031 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:51,031 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 7 2024-07-04 00:16:51,031 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:51,031 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 7 2024-07-04 00:16:51,880 INFO Thread-12 :152382 [dir_watcher.py:_on_file_created():271] file/dir created: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/wandb-summary.json 2024-07-04 00:16:52,018 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: poll_exit 2024-07-04 00:16:53,089 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 8 2024-07-04 00:16:53,089 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: poll_exit 2024-07-04 00:16:53,089 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:53,090 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 8 2024-07-04 00:16:53,090 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:53,090 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 8 2024-07-04 00:16:53,090 INFO SenderThread:152382 [job_builder.py:build():440] Attempting to build job artifact 2024-07-04 00:16:53,092 INFO SenderThread:152382 [job_builder.py:_get_source_type():569] is repo sourced job 2024-07-04 00:16:53,092 WARNING SenderThread:152382 [job_builder.py:_log_if_verbose():274] No program path found, not creating job artifact. See https://docs.wandb.ai/guides/launch/create-job 2024-07-04 00:16:53,092 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 9 2024-07-04 00:16:53,092 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:53,092 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 9 2024-07-04 00:16:53,092 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:53,092 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 9 2024-07-04 00:16:53,092 INFO SenderThread:152382 [dir_watcher.py:finish():358] shutting down directory watcher 2024-07-04 00:16:53,881 INFO SenderThread:152382 [dir_watcher.py:_on_file_modified():288] file/dir modified: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/output.log 2024-07-04 00:16:53,882 INFO SenderThread:152382 [dir_watcher.py:finish():388] scan: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files 2024-07-04 00:16:53,882 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/conda-environment.yaml conda-environment.yaml 2024-07-04 00:16:53,882 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/config.yaml config.yaml 2024-07-04 00:16:53,885 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/wandb-metadata.json wandb-metadata.json 2024-07-04 00:16:53,885 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/requirements.txt requirements.txt 2024-07-04 00:16:53,887 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/output.log output.log 2024-07-04 00:16:53,888 INFO SenderThread:152382 [dir_watcher.py:finish():402] scan save: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/wandb-summary.json wandb-summary.json 2024-07-04 00:16:53,890 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 10 2024-07-04 00:16:53,892 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:53,892 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 10 2024-07-04 00:16:53,894 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:53,894 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 10 2024-07-04 00:16:53,894 INFO SenderThread:152382 [file_pusher.py:finish():169] shutting down file pusher 2024-07-04 00:16:54,018 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: poll_exit 2024-07-04 00:16:54,018 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: poll_exit 2024-07-04 00:16:54,416 INFO wandb-upload_4:152382 [upload_job.py:push():130] Uploaded file /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/wandb-summary.json 2024-07-04 00:16:54,417 INFO wandb-upload_0:152382 [upload_job.py:push():130] Uploaded file /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/conda-environment.yaml 2024-07-04 00:16:54,425 INFO wandb-upload_2:152382 [upload_job.py:push():130] Uploaded file /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/requirements.txt 2024-07-04 00:16:54,438 INFO wandb-upload_1:152382 [upload_job.py:push():130] Uploaded file /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/config.yaml 2024-07-04 00:16:54,462 INFO wandb-upload_3:152382 [upload_job.py:push():130] Uploaded file /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/files/output.log 2024-07-04 00:16:54,662 INFO Thread-11 :152382 [sender.py:transition_state():614] send defer: 11 2024-07-04 00:16:54,662 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:54,662 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 11 2024-07-04 00:16:54,662 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:54,663 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 11 2024-07-04 00:16:54,663 INFO SenderThread:152382 [file_pusher.py:join():175] waiting for file pusher 2024-07-04 00:16:54,663 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 12 2024-07-04 00:16:54,663 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:54,663 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 12 2024-07-04 00:16:54,663 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:54,663 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 12 2024-07-04 00:16:54,663 INFO SenderThread:152382 [file_stream.py:finish():601] file stream finish called 2024-07-04 00:16:55,019 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: poll_exit 2024-07-04 00:16:55,201 INFO SenderThread:152382 [file_stream.py:finish():605] file stream finish is done 2024-07-04 00:16:55,201 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 13 2024-07-04 00:16:55,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: poll_exit 2024-07-04 00:16:55,201 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:55,201 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 13 2024-07-04 00:16:55,201 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:55,201 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 13 2024-07-04 00:16:55,201 INFO SenderThread:152382 [sender.py:transition_state():614] send defer: 14 2024-07-04 00:16:55,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: defer 2024-07-04 00:16:55,202 INFO HandlerThread:152382 [handler.py:handle_request_defer():184] handle defer: 14 2024-07-04 00:16:55,202 DEBUG SenderThread:152382 [sender.py:send():379] send: final 2024-07-04 00:16:55,202 DEBUG SenderThread:152382 [sender.py:send():379] send: footer 2024-07-04 00:16:55,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: defer 2024-07-04 00:16:55,202 INFO SenderThread:152382 [sender.py:send_request_defer():610] handle sender defer: 14 2024-07-04 00:16:55,202 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: poll_exit 2024-07-04 00:16:55,202 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: poll_exit 2024-07-04 00:16:55,203 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: server_info 2024-07-04 00:16:55,203 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: server_info 2024-07-04 00:16:55,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: poll_exit 2024-07-04 00:16:55,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: get_summary 2024-07-04 00:16:55,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: sampled_history 2024-07-04 00:16:55,205 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: internal_messages 2024-07-04 00:16:55,370 DEBUG SenderThread:152382 [sender.py:send_request():406] send_request: poll_exit 2024-07-04 00:16:55,370 INFO MainThread:152382 [wandb_run.py:_footer_history_summary_info():3991] rendering history 2024-07-04 00:16:55,370 INFO MainThread:152382 [wandb_run.py:_footer_history_summary_info():4023] rendering summary 2024-07-04 00:16:55,370 INFO MainThread:152382 [wandb_run.py:_footer_sync_info():3950] logging synced files 2024-07-04 00:16:55,371 DEBUG HandlerThread:152382 [handler.py:handle_request():158] handle_request: shutdown 2024-07-04 00:16:55,371 INFO HandlerThread:152382 [handler.py:finish():882] shutting down handler 2024-07-04 00:16:56,205 INFO WriterThread:152382 [datastore.py:close():296] close: /system/user/studentwork/mohammed/DM/pokemon/diffusers/wandb/run-20240703_232113-0z9lks5c/run-0z9lks5c.wandb 2024-07-04 00:16:56,370 INFO SenderThread:152382 [sender.py:finish():1608] shutting down sender 2024-07-04 00:16:56,370 INFO SenderThread:152382 [file_pusher.py:finish():169] shutting down file pusher 2024-07-04 00:16:56,370 INFO SenderThread:152382 [file_pusher.py:join():175] waiting for file pusher