2024-12-17 22:17:07,590 INFO StreamThr :84579 [internal.py:wandb_internal():85] W&B internal server running at pid: 84579, started at: 2024-12-17 22:17:07.589593 2024-12-17 22:17:07,591 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status 2024-12-17 22:17:07,601 INFO WriterThread:84579 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/run-sqbcnf7o.wandb 2024-12-17 22:17:07,601 DEBUG SenderThread:84579 [sender.py:send():379] send: header 2024-12-17 22:17:07,608 DEBUG SenderThread:84579 [sender.py:send():379] send: run 2024-12-17 22:17:07,901 INFO SenderThread:84579 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files 2024-12-17 22:17:07,901 INFO SenderThread:84579 [sender.py:_start_run_threads():1188] run started: sqbcnf7o with start time 1734491827.589933 2024-12-17 22:17:07,920 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: check_version 2024-12-17 22:17:07,920 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: check_version 2024-12-17 22:17:07,967 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: run_start 2024-12-17 22:17:07,972 DEBUG HandlerThread:84579 [system_info.py:__init__():26] System info init 2024-12-17 22:17:07,972 DEBUG HandlerThread:84579 [system_info.py:__init__():41] System info init done 2024-12-17 22:17:07,972 INFO HandlerThread:84579 [system_monitor.py:start():194] Starting system monitor 2024-12-17 22:17:07,972 INFO SystemMonitor:84579 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-17 22:17:07,972 INFO HandlerThread:84579 [system_monitor.py:probe():214] Collecting system info 2024-12-17 22:17:07,973 INFO SystemMonitor:84579 [interfaces.py:start():188] Started cpu monitoring 2024-12-17 22:17:07,974 INFO SystemMonitor:84579 [interfaces.py:start():188] Started disk monitoring 2024-12-17 22:17:07,974 INFO SystemMonitor:84579 [interfaces.py:start():188] Started gpu monitoring 2024-12-17 22:17:07,974 INFO SystemMonitor:84579 [interfaces.py:start():188] Started memory monitoring 2024-12-17 22:17:07,975 INFO SystemMonitor:84579 [interfaces.py:start():188] Started network monitoring 2024-12-17 22:17:07,984 DEBUG HandlerThread:84579 [system_info.py:probe():152] Probing system 2024-12-17 22:17:07,986 DEBUG HandlerThread:84579 [system_info.py:_probe_git():137] Probing git 2024-12-17 22:17:07,991 DEBUG HandlerThread:84579 [system_info.py:_probe_git():145] Probing git done 2024-12-17 22:17:07,991 DEBUG HandlerThread:84579 [system_info.py:probe():200] Probing system done 2024-12-17 22:17:07,991 DEBUG HandlerThread:84579 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T03:17:07.984722', 'startedAt': '2024-12-18T03:17:07.579033', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3744.56375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.353, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.033, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 416.7548141479492}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2024-12-17 22:17:07,991 INFO HandlerThread:84579 [system_monitor.py:probe():224] Finished collecting system info 2024-12-17 22:17:07,991 INFO HandlerThread:84579 [system_monitor.py:probe():227] Publishing system info 2024-12-17 22:17:07,991 DEBUG HandlerThread:84579 [system_info.py:_save_code():44] Saving code 2024-12-17 22:17:07,993 DEBUG HandlerThread:84579 [system_info.py:_save_code():67] Saving code done 2024-12-17 22:17:07,993 DEBUG HandlerThread:84579 [system_info.py:_save_patches():84] Saving git patches 2024-12-17 22:17:08,065 DEBUG HandlerThread:84579 [system_info.py:_save_patches():126] Saving git patches done 2024-12-17 22:17:08,065 INFO HandlerThread:84579 [system_monitor.py:probe():229] Finished publishing system info 2024-12-17 22:17:08,068 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:17:08,068 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-17 22:17:08,069 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2024-12-17 22:17:08,069 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file diff.patch with policy now 2024-12-17 22:17:08,301 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: python_packages 2024-12-17 22:17:08,301 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:17:08,301 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: python_packages 2024-12-17 22:17:08,302 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:17:08,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:08,313 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/eglwkdot-wandb-metadata.json 2024-12-17 22:17:08,323 INFO wandb-upload_2:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/7k7jg9di-diff.patch 2024-12-17 22:17:08,349 INFO HandlerThread:84579 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/tb" save: true } control { relay_id: "81d3cf2cbfed43cbb7d230993cc9c89e" } _info { stream_id: "sqbcnf7o" } 2024-12-17 22:17:08,349 DEBUG HandlerThread:84579 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2024-12-17 22:17:08,395 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/ggysc832-code/FISH/eval_robot.py 2024-12-17 22:17:08,456 DEBUG SenderThread:84579 [sender.py:send():379] send: telemetry 2024-12-17 22:17:08,456 DEBUG SenderThread:84579 [sender.py:send():379] send: tbrecord 2024-12-17 22:17:08,456 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:17:08,457 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 with policy live 2024-12-17 22:17:08,631 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/hghsffpo-events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:17:08,902 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/diff.patch 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/code/FISH/eval_robot.py 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-metadata.json 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/requirements.txt 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/code/FISH 2024-12-17 22:17:08,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/code 2024-12-17 22:17:09,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:10,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:10,902 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:11,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:12,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:13,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:13,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:13,845 DEBUG SenderThread:84579 [sender.py:send():379] send: config 2024-12-17 22:17:14,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:14,903 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:15,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:16,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:17,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:18,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:18,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:18,904 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:19,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:20,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:20,904 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:21,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:22,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:23,301 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:17:23,301 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:17:23,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:23,434 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:24,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:24,906 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:25,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:26,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:26,906 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:27,304 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:28,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:28,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:29,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:30,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:30,907 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:31,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:32,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:32,908 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:33,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:34,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:34,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:35,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:36,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:36,909 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:37,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:38,301 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:17:38,302 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:17:38,345 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:39,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:40,117 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:40,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:40,910 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:40,910 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/config.yaml 2024-12-17 22:17:41,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:42,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:43,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:44,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:44,911 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:45,303 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:45,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:46,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:47,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:48,307 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:49,308 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:50,308 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:51,113 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:51,308 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:52,310 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:52,913 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:53,301 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:17:53,302 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:17:53,345 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:54,310 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:55,329 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:56,211 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:17:56,225 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:17:56,225 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 with policy now 2024-12-17 22:17:56,225 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:17:56,226 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:17:56,226 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:17:56,228 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:17:56,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:56,485 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/uq7df1o8-media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval 2024-12-17 22:17:56,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos 2024-12-17 22:17:57,186 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/ucxh4z2j-media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 2024-12-17 22:17:57,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:58,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:17:58,915 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:17:59,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:00,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:00,916 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:01,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:01,477 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:02,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:03,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:04,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:04,917 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:18:04,918 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:05,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:06,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:06,918 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:07,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:07,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:07,975 DEBUG SystemMonitor:84579 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-17 22:18:07,976 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:18:08,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:18:08,302 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:18:08,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:08,919 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:09,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:10,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:10,920 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:11,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:12,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:12,370 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:13,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:14,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:15,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:16,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:17,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:18,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:18,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:19,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:20,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:21,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:22,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:23,302 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:18:23,302 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:18:23,345 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:23,435 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:24,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:25,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:26,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:27,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:28,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:29,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:29,314 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:30,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:31,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:32,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:33,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:34,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:34,717 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:35,315 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:35,940 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:18:35,940 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 with policy now 2024-12-17 22:18:35,942 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:18:35,943 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:18:35,943 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:18:35,944 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:18:36,188 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/4cgqz4qo-media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 2024-12-17 22:18:36,319 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:36,928 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:36,928 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:18:36,928 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 2024-12-17 22:18:36,928 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval 2024-12-17 22:18:37,214 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/9sjtqb51-media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 2024-12-17 22:18:37,318 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:37,976 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:18:38,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:18:38,307 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:18:38,349 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:39,332 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:39,791 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:40,335 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:40,929 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:41,338 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:42,335 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:43,336 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:44,339 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:44,930 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:45,309 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:45,337 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:46,336 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:47,335 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:48,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:49,350 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:50,328 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:50,340 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:51,343 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:52,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:52,739 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:18:52,739 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:18:52,740 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:18:52,933 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:18:52,933 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:18:53,180 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/qz8moi18-events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:18:53,305 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:18:53,305 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:18:53,340 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:54,353 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:54,934 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:55,356 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:55,397 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:18:56,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:56,934 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:57,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:58,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:18:58,935 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:18:59,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:00,349 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:01,311 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:01,349 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:02,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:03,348 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:04,350 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:05,349 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:06,312 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:06,350 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:07,351 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:07,976 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:19:08,306 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:19:08,307 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:19:08,350 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:09,350 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:10,351 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:11,351 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:11,381 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:12,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:13,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:14,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:15,352 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:16,353 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:17,313 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:17,353 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:18,353 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:19,353 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:20,354 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:21,354 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:22,357 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:23,310 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:19:23,310 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:19:23,357 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:23,385 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:24,355 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:25,368 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:25,684 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:19:25,684 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 with policy now 2024-12-17 22:19:25,689 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:19:25,690 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:19:25,690 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:19:25,691 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:19:25,933 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/qyjuq5lx-media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 2024-12-17 22:19:25,944 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:19:25,944 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 2024-12-17 22:19:25,944 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval 2024-12-17 22:19:26,186 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/e6gxg7js-media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 2024-12-17 22:19:26,367 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:26,944 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:27,375 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:28,377 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:29,316 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:29,382 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:30,379 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:30,946 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:31,379 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:32,413 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:33,390 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:34,389 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:34,947 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:35,316 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:35,391 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:36,398 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:36,589 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:19:36,589 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:19:36,590 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:19:36,948 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:19:36,948 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:19:37,394 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:37,977 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:19:38,351 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:19:38,352 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:19:38,401 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:38,948 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:39,407 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:40,403 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:40,949 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:41,045 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:41,408 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:42,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:43,416 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:44,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:44,950 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:19:45,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:46,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:46,505 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:47,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:48,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:49,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:50,399 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:51,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:52,316 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:52,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:53,323 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:19:53,323 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:19:53,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:54,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:55,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:56,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:57,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:57,427 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:19:58,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:19:59,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:00,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:01,400 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:02,401 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:03,316 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:03,401 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:04,404 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:05,402 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:06,402 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:07,402 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:07,977 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:20:08,316 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:08,323 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:20:08,323 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:20:08,402 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:09,403 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:10,403 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:10,959 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:11,130 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:20:11,130 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 with policy now 2024-12-17 22:20:11,135 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:20:11,135 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:20:11,135 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:20:11,136 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:20:11,398 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/6c53q71e-media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 2024-12-17 22:20:11,416 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:11,959 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:20:11,959 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 2024-12-17 22:20:11,960 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval 2024-12-17 22:20:12,225 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/g46ird7n-media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 2024-12-17 22:20:12,408 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:12,960 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:13,327 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:13,415 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:14,427 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:15,426 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:16,424 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:16,961 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:17,424 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:18,428 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:18,962 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:19,322 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:19,464 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:20,433 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:21,435 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:22,435 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:23,328 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:20:23,329 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:20:23,444 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:24,406 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:24,450 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:25,449 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:26,461 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:27,464 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:27,673 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:20:27,673 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:20:27,673 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:20:27,965 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:20:27,965 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:20:28,200 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/hjkbjwos-events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:20:28,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:28,965 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:29,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:30,336 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:30,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:30,966 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:31,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:32,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:32,966 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:33,467 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:34,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:34,967 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:35,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:36,162 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:36,457 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:36,968 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:37,458 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:37,978 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:20:38,328 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:20:38,328 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:20:38,458 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:39,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:40,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:41,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:41,472 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:42,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:43,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:44,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:45,460 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:46,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:47,333 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:47,459 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:48,460 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:49,461 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:50,461 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:51,461 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:52,333 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:52,461 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:53,331 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:20:53,331 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:20:53,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:54,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:55,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:56,462 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:57,470 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:57,505 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:20:58,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:20:58,975 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:20:59,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:00,464 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:01,463 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:02,493 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:02,529 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:21:02,795 DEBUG SenderThread:84579 [sender.py:send():379] send: files 2024-12-17 22:21:02,795 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 with policy now 2024-12-17 22:21:02,798 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:21:02,798 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:21:02,799 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:21:02,799 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:21:02,976 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:02,976 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:21:02,977 INFO Thread-12 :84579 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 2024-12-17 22:21:02,977 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval 2024-12-17 22:21:03,064 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/jpil4fgf-media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 2024-12-17 22:21:03,333 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/qluoymh3-media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 2024-12-17 22:21:03,488 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:04,481 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:04,977 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:05,487 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:06,490 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:07,482 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:07,701 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:21:07,978 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:21:08,354 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:21:08,354 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:21:08,483 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:08,979 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:09,481 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:10,489 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:10,980 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:11,453 DEBUG SenderThread:84579 [sender.py:send():379] send: exit 2024-12-17 22:21:11,454 INFO SenderThread:84579 [sender.py:send_exit():586] handling exit code: 0 2024-12-17 22:21:11,454 INFO SenderThread:84579 [sender.py:send_exit():588] handling runtime: 243 2024-12-17 22:21:11,454 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:21:11,455 INFO SenderThread:84579 [sender.py:send_exit():594] send defer 2024-12-17 22:21:11,455 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:11,455 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-17 22:21:11,455 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:11,455 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-17 22:21:11,455 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 1 2024-12-17 22:21:11,455 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:11,455 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-17 22:21:11,456 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:11,456 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-17 22:21:11,456 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 2 2024-12-17 22:21:11,456 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:11,456 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-17 22:21:11,456 INFO HandlerThread:84579 [system_monitor.py:finish():203] Stopping system monitor 2024-12-17 22:21:11,456 DEBUG SystemMonitor:84579 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-17 22:21:11,456 INFO HandlerThread:84579 [interfaces.py:finish():200] Joined cpu monitor 2024-12-17 22:21:11,456 DEBUG SystemMonitor:84579 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-17 22:21:11,456 INFO HandlerThread:84579 [interfaces.py:finish():200] Joined disk monitor 2024-12-17 22:21:11,502 INFO HandlerThread:84579 [interfaces.py:finish():200] Joined gpu monitor 2024-12-17 22:21:11,502 INFO HandlerThread:84579 [interfaces.py:finish():200] Joined memory monitor 2024-12-17 22:21:11,502 INFO HandlerThread:84579 [interfaces.py:finish():200] Joined network monitor 2024-12-17 22:21:11,503 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:11,503 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-17 22:21:11,503 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 3 2024-12-17 22:21:11,503 DEBUG SenderThread:84579 [sender.py:send():379] send: stats 2024-12-17 22:21:11,503 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:11,503 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-17 22:21:11,503 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:11,503 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-17 22:21:11,503 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 4 2024-12-17 22:21:11,503 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:11,503 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-17 22:21:11,980 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:21:11,980 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:21:12,981 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:17,505 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:17,505 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:21:17,505 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:17,506 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: keepalive 2024-12-17 22:21:17,506 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-17 22:21:17,506 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 5 2024-12-17 22:21:17,506 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:21:17,506 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:17,506 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:21:17,506 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-17 22:21:17,507 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:21:17,507 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:17,507 DEBUG SenderThread:84579 [sender.py:send():379] send: history 2024-12-17 22:21:17,507 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:21:17,511 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:21:17,511 DEBUG SenderThread:84579 [sender.py:send():379] send: summary 2024-12-17 22:21:17,511 INFO SenderThread:84579 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:21:17,511 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:17,512 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-17 22:21:17,512 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 6 2024-12-17 22:21:17,512 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:17,512 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-17 22:21:17,512 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:17,512 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-17 22:21:17,512 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 7 2024-12-17 22:21:17,512 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:21:17,512 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:17,512 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-17 22:21:17,512 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:17,513 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-17 22:21:17,982 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:21:18,454 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:18,708 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 8 2024-12-17 22:21:18,708 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:18,708 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:18,709 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-17 22:21:18,709 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:18,709 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-17 22:21:18,709 INFO SenderThread:84579 [job_builder.py:build():440] Attempting to build job artifact 2024-12-17 22:21:18,715 INFO SenderThread:84579 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-17 22:21:18,731 INFO SenderThread:84579 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-17 22:21:18,743 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 9 2024-12-17 22:21:18,743 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:18,743 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-17 22:21:18,743 DEBUG SenderThread:84579 [sender.py:send():379] send: artifact 2024-12-17 22:21:18,983 INFO Thread-12 :84579 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:19,455 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:19,527 INFO wandb-upload_2:84579 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpqschr5qi 2024-12-17 22:21:19,527 INFO wandb-upload_0:84579 [upload_job.py:push():85] Skipped uploading /tmp/tmpsbqi9vuh/wandb-job.json 2024-12-17 22:21:19,649 INFO wandb-upload_1:84579 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpfksjk6nr 2024-12-17 22:21:20,210 INFO SenderThread:84579 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzQxMDY4OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzM4ODU2OA==', 'versionIndex': 383}}} 2024-12-17 22:21:20,210 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:20,210 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-17 22:21:20,210 INFO SenderThread:84579 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-17 22:21:20,984 INFO SenderThread:84579 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files 2024-12-17 22:21:20,984 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/config.yaml config.yaml 2024-12-17 22:21:20,984 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json wandb-summary.json 2024-12-17 22:21:20,984 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/requirements.txt requirements.txt 2024-12-17 22:21:20,984 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:21:20,987 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-metadata.json wandb-metadata.json 2024-12-17 22:21:20,987 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log output.log 2024-12-17 22:21:20,987 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/diff.patch diff.patch 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 media/videos/eval/2_eval_2_312f8f530abaf399946b.mp4 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 media/videos/eval/0_eval_0_886132ad98d0c31f9311.mp4 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 media/videos/eval/3_eval_3_91b538bdbc0a7862dd54.mp4 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 media/videos/eval/4_eval_4_fdb67280e4956e7c7ab4.mp4 2024-12-17 22:21:20,989 INFO SenderThread:84579 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 media/videos/eval/1_eval_1_c21bfe5dd150e694f7bd.mp4 2024-12-17 22:21:20,989 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 10 2024-12-17 22:21:20,989 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:20,989 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:20,990 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-17 22:21:20,992 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:20,992 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-17 22:21:20,992 INFO SenderThread:84579 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 22:21:21,167 INFO wandb-upload_0:84579 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/wandb-summary.json 2024-12-17 22:21:21,200 INFO wandb-upload_1:84579 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/requirements.txt 2024-12-17 22:21:21,217 INFO wandb-upload_2:84579 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/config.yaml 2024-12-17 22:21:21,265 INFO wandb-upload_3:84579 [upload_job.py:push():130] Uploaded file /tmp/tmpf6cm3vldwandb/r373gfgp-events.out.tfevents.1734491828.leonmkim-ROG-Strix-G15CS-G15CS.84391.0 2024-12-17 22:21:21,320 INFO wandb-upload_4:84579 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/files/output.log 2024-12-17 22:21:21,456 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:21,456 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:21,521 INFO Thread-11 (_thread_body):84579 [sender.py:transition_state():614] send defer: 11 2024-12-17 22:21:21,521 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:21,521 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-17 22:21:21,521 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:21,521 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-17 22:21:21,521 INFO SenderThread:84579 [file_pusher.py:join():175] waiting for file pusher 2024-12-17 22:21:21,522 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 12 2024-12-17 22:21:21,522 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:21,522 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-17 22:21:21,522 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:21,523 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-17 22:21:21,523 INFO SenderThread:84579 [file_stream.py:finish():601] file stream finish called 2024-12-17 22:21:21,596 INFO SenderThread:84579 [file_stream.py:finish():605] file stream finish is done 2024-12-17 22:21:21,596 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 13 2024-12-17 22:21:21,597 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:21,597 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-17 22:21:21,597 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:21,597 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-17 22:21:21,597 INFO SenderThread:84579 [sender.py:transition_state():614] send defer: 14 2024-12-17 22:21:21,597 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:21:21,597 DEBUG SenderThread:84579 [sender.py:send():379] send: final 2024-12-17 22:21:21,597 INFO HandlerThread:84579 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-17 22:21:21,597 DEBUG SenderThread:84579 [sender.py:send():379] send: footer 2024-12-17 22:21:21,597 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: defer 2024-12-17 22:21:21,598 INFO SenderThread:84579 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-17 22:21:21,598 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:21,598 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:21,598 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: server_info 2024-12-17 22:21:21,598 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: server_info 2024-12-17 22:21:21,600 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:21:21,600 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: get_summary 2024-12-17 22:21:21,600 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-17 22:21:21,600 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:21:21,661 DEBUG SenderThread:84579 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:21:21,662 INFO MainThread:84579 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2024-12-17 22:21:21,662 INFO MainThread:84579 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2024-12-17 22:21:21,662 INFO MainThread:84579 [wandb_run.py:_footer_sync_info():3975] logging synced files 2024-12-17 22:21:21,662 DEBUG HandlerThread:84579 [handler.py:handle_request():158] handle_request: shutdown 2024-12-17 22:21:21,662 INFO HandlerThread:84579 [handler.py:finish():882] shutting down handler 2024-12-17 22:21:22,600 INFO WriterThread:84579 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/511_0/221701/wandb/run-20241217_221707-sqbcnf7o/run-sqbcnf7o.wandb 2024-12-17 22:21:22,662 INFO SenderThread:84579 [sender.py:finish():1615] shutting down sender 2024-12-17 22:21:22,662 INFO SenderThread:84579 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 22:21:22,662 INFO SenderThread:84579 [file_pusher.py:join():175] waiting for file pusher