2025-01-07 20:11:20,661 INFO StreamThr :2221501 [internal.py:wandb_internal():85] W&B internal server running at pid: 2221501, started at: 2025-01-07 20:11:20.660615 2025-01-07 20:11:20,661 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status 2025-01-07 20:11:20,675 INFO WriterThread:2221501 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/run-urh96fgp.wandb 2025-01-07 20:11:20,675 DEBUG SenderThread:2221501 [sender.py:send():379] send: header 2025-01-07 20:11:20,688 DEBUG SenderThread:2221501 [sender.py:send():379] send: run 2025-01-07 20:11:20,894 INFO SenderThread:2221501 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files 2025-01-07 20:11:20,894 INFO SenderThread:2221501 [sender.py:_start_run_threads():1188] run started: urh96fgp with start time 1736298680.662175 2025-01-07 20:11:20,911 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: check_version 2025-01-07 20:11:20,912 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: check_version 2025-01-07 20:11:20,955 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: run_start 2025-01-07 20:11:20,960 DEBUG HandlerThread:2221501 [system_info.py:__init__():26] System info init 2025-01-07 20:11:20,960 DEBUG HandlerThread:2221501 [system_info.py:__init__():41] System info init done 2025-01-07 20:11:20,960 INFO HandlerThread:2221501 [system_monitor.py:start():194] Starting system monitor 2025-01-07 20:11:20,960 INFO SystemMonitor:2221501 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-07 20:11:20,960 INFO HandlerThread:2221501 [system_monitor.py:probe():214] Collecting system info 2025-01-07 20:11:20,961 INFO SystemMonitor:2221501 [interfaces.py:start():188] Started cpu monitoring 2025-01-07 20:11:20,961 INFO SystemMonitor:2221501 [interfaces.py:start():188] Started disk monitoring 2025-01-07 20:11:20,962 INFO SystemMonitor:2221501 [interfaces.py:start():188] Started gpu monitoring 2025-01-07 20:11:20,962 INFO SystemMonitor:2221501 [interfaces.py:start():188] Started memory monitoring 2025-01-07 20:11:20,963 INFO SystemMonitor:2221501 [interfaces.py:start():188] Started network monitoring 2025-01-07 20:11:20,973 DEBUG HandlerThread:2221501 [system_info.py:probe():152] Probing system 2025-01-07 20:11:20,974 DEBUG HandlerThread:2221501 [system_info.py:_probe_git():137] Probing git 2025-01-07 20:11:20,979 DEBUG HandlerThread:2221501 [system_info.py:_probe_git():145] Probing git done 2025-01-07 20:11:20,979 DEBUG HandlerThread:2221501 [system_info.py:probe():200] Probing system done 2025-01-07 20:11:20,979 DEBUG HandlerThread:2221501 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T01:11:20.973511', 'startedAt': '2025-01-08T01:11:20.653238', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3560.282, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4466.498, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.02, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.992, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.002, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 554.1286163330078}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-07 20:11:20,979 INFO HandlerThread:2221501 [system_monitor.py:probe():224] Finished collecting system info 2025-01-07 20:11:20,979 INFO HandlerThread:2221501 [system_monitor.py:probe():227] Publishing system info 2025-01-07 20:11:20,979 DEBUG HandlerThread:2221501 [system_info.py:_save_code():44] Saving code 2025-01-07 20:11:20,981 DEBUG HandlerThread:2221501 [system_info.py:_save_code():67] Saving code done 2025-01-07 20:11:20,981 DEBUG HandlerThread:2221501 [system_info.py:_save_patches():84] Saving git patches 2025-01-07 20:11:21,010 DEBUG HandlerThread:2221501 [system_info.py:_save_patches():126] Saving git patches done 2025-01-07 20:11:21,010 INFO HandlerThread:2221501 [system_monitor.py:probe():229] Finished publishing system info 2025-01-07 20:11:21,014 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:11:21,014 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-07 20:11:21,014 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-07 20:11:21,267 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: python_packages 2025-01-07 20:11:21,267 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: python_packages 2025-01-07 20:11:21,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:21,269 DEBUG SenderThread:2221501 [sender.py:send():379] send: telemetry 2025-01-07 20:11:21,289 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/rxji41gk-wandb-metadata.json 2025-01-07 20:11:21,293 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/f4kmqkjd-code/FISH/eval_robot.py 2025-01-07 20:11:21,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:11:21,309 INFO HandlerThread:2221501 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/tb" save: true } control { relay_id: "b30f97c685eb4f2885a4012e0da1b4ce" } _info { stream_id: "urh96fgp" } 2025-01-07 20:11:21,309 DEBUG HandlerThread:2221501 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-07 20:11:21,310 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:11:21,448 DEBUG SenderThread:2221501 [sender.py:send():379] send: tbrecord 2025-01-07 20:11:21,449 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:11:21,449 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 with policy live 2025-01-07 20:11:21,641 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/d8m4ti26-events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:11:21,896 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/requirements.txt 2025-01-07 20:11:21,896 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:11:21,896 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:21,897 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/code/FISH/eval_robot.py 2025-01-07 20:11:21,897 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-metadata.json 2025-01-07 20:11:21,897 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/code/FISH 2025-01-07 20:11:21,897 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/code 2025-01-07 20:11:22,267 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:23,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:23,895 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:24,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:25,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:25,533 DEBUG SenderThread:2221501 [sender.py:send():379] send: config 2025-01-07 20:11:26,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:26,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:27,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:27,897 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:28,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:29,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:30,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:31,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:31,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:31,898 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:32,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:33,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:34,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:35,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:35,899 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:36,267 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:11:36,267 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:11:36,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:36,330 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:37,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:38,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:39,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:39,900 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:40,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:41,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:41,494 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:42,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:43,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:43,901 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:11:44,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:45,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:46,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:47,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:47,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:48,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:49,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:50,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:51,267 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:11:51,268 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:11:51,313 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:52,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:52,406 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:52,903 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/config.yaml 2025-01-07 20:11:53,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:54,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:55,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:56,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:57,274 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:58,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:11:58,274 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:59,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:11:59,905 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:00,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:01,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:02,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:03,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:04,269 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:04,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:05,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:06,267 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:12:06,268 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:12:06,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:07,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:08,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:08,955 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:12:08,955 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:12:08,955 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 with policy now 2025-01-07 20:12:08,956 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:12:08,956 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:12:08,958 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:12:09,206 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/tqh7i2v8-media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 2025-01-07 20:12:09,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media 2025-01-07 20:12:09,908 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos 2025-01-07 20:12:09,967 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:10,213 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/gyo0vgzq-media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 2025-01-07 20:12:10,278 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:11,278 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:12,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:13,278 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:13,910 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:14,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:15,185 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:15,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:16,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:17,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:17,911 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:18,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:19,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:20,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:20,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:20,963 DEBUG SystemMonitor:2221501 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-07 20:12:20,963 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:12:21,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:12:21,268 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:12:21,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:22,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:23,279 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:24,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:25,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:25,352 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:26,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:27,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:28,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:29,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:29,915 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:12:30,177 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/06kf33l8-events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:12:30,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:30,465 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:31,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:31,916 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:32,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:33,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:34,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:35,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:35,466 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:35,917 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:36,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:12:36,268 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:12:36,313 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:37,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:38,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:39,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:40,281 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:41,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:41,282 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:42,282 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:43,282 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:43,920 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:44,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:45,283 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:46,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:46,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:47,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:48,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:49,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:50,285 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:50,964 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:12:51,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:12:51,268 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:12:51,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:51,415 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:52,285 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:53,285 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:53,923 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:12:54,285 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:55,293 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:56,293 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:57,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:12:57,293 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:58,293 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:12:59,294 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:00,297 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:00,457 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:13:00,457 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 with policy now 2025-01-07 20:13:00,460 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:13:00,461 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:13:00,461 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:13:00,461 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:13:00,699 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/hz4ko86e-media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 2025-01-07 20:13:00,925 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:13:00,925 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 2025-01-07 20:13:00,925 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval 2025-01-07 20:13:01,198 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/m3177g2j-media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 2025-01-07 20:13:01,302 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:01,925 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:02,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:02,302 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:02,925 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:03,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:04,299 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:05,299 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:05,926 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:06,277 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:13:06,277 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:13:06,321 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:07,298 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:07,927 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:08,005 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:08,299 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:09,299 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:10,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:11,299 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:12,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:13,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:13,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:13,929 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:13:14,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:14,716 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:13:14,716 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:13:14,716 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:13:14,929 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:13:15,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:15,930 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:16,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:17,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:18,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:18,697 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:19,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:19,931 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:20,300 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:20,965 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:13:21,268 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:13:21,269 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:13:21,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:22,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:23,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:24,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:24,357 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:25,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:26,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:27,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:28,301 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:29,302 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:30,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:30,302 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:31,302 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:32,305 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:33,278 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:13:33,278 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 with policy now 2025-01-07 20:13:33,280 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:13:33,281 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:13:33,281 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:13:33,282 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:13:33,307 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:33,553 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/ur6qagdb-media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 2025-01-07 20:13:33,935 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:13:33,936 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:33,936 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 2025-01-07 20:13:33,936 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval 2025-01-07 20:13:34,197 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/xh7y27ox-media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 2025-01-07 20:13:34,310 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:34,936 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:35,305 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:35,309 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:35,936 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:36,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:13:36,276 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:13:36,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:37,321 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:37,937 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:38,319 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:39,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:40,315 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:40,761 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:41,315 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:41,938 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:42,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:43,314 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:44,314 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:45,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:46,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:46,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:47,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:48,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:49,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:50,315 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:50,539 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:13:50,539 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:13:50,550 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:13:50,941 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:13:50,942 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:13:50,965 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:13:51,224 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/uev3j49l-events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:13:51,305 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:13:51,305 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:51,306 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:13:51,325 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:51,942 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:52,322 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:53,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:53,942 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:54,315 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:55,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:55,943 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:13:56,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:57,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:13:57,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:58,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:13:59,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:00,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:01,316 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:01,945 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:02,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:03,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:03,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:04,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:05,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:06,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:14:06,270 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:14:06,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:07,317 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:08,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:08,330 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:09,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:10,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:11,318 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:12,319 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:13,319 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:14,273 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:14,319 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:15,320 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:16,320 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:17,320 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:18,320 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:19,275 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:19,321 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:20,321 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:20,965 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:14:21,270 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:14:21,270 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:14:21,321 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:22,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:23,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:24,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:24,364 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:25,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:26,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:27,323 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:28,324 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:29,333 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:29,568 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:29,955 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:30,076 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:14:30,076 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 with policy now 2025-01-07 20:14:30,082 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:14:30,083 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:14:30,083 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:14:30,088 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:14:30,328 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:30,347 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/m8z85ohs-media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 2025-01-07 20:14:30,956 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:14:30,956 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 2025-01-07 20:14:30,956 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval 2025-01-07 20:14:31,245 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/zl5hyo3l-media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 2025-01-07 20:14:31,332 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:31,956 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:32,326 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:33,331 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:34,331 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:34,689 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:35,332 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:35,957 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:36,284 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:14:36,284 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:14:36,333 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:37,334 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:37,958 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:38,340 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:39,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:40,336 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:40,434 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:41,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:42,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:43,337 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:43,603 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:14:43,603 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:14:43,604 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:14:43,960 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:14:43,960 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:43,960 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:14:44,339 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:45,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:45,961 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:46,157 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:46,347 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:47,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:47,961 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:48,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:49,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:50,341 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:50,966 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:14:51,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:14:51,271 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:14:51,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:51,385 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:52,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:53,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:54,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:55,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:55,964 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:14:56,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:57,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:14:57,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:58,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:14:59,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:00,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:01,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:02,276 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:02,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:03,344 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:04,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:05,344 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:06,271 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:15:06,271 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:15:06,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:07,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:07,420 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:08,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:09,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:10,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:11,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:12,345 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:13,277 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:13,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:14,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:15,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:16,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:17,346 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:18,347 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:18,842 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:19,351 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:19,972 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:20,224 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:15:20,224 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 with policy now 2025-01-07 20:15:20,228 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:15:20,228 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:15:20,229 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:20,229 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:20,349 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:20,500 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/dubhrrmg-media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 2025-01-07 20:15:20,972 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:20,973 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 2025-01-07 20:15:20,973 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval 2025-01-07 20:15:21,006 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:15:21,291 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:15:21,291 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:15:21,297 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/mnpvnpde-media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 2025-01-07 20:15:21,355 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:21,973 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:22,361 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:22,973 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:23,357 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:23,977 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:24,356 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:25,358 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:25,974 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:26,364 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:27,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:27,975 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:28,375 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:29,362 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:29,837 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:30,365 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:31,364 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:31,628 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 20:15:31,628 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 20:15:31,803 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:15:31,976 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:31,976 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:15:31,976 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 20:15:31,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media 2025-01-07 20:15:31,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table 2025-01-07 20:15:31,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval 2025-01-07 20:15:32,155 INFO wandb-upload_1:2221501 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpezyg0hgw 2025-01-07 20:15:32,188 INFO wandb-upload_2:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/bkk4uymf-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 20:15:32,194 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 20:15:32,204 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 20:15:32,261 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/sp33j6e1-events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:15:32,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:32,551 INFO SenderThread:2221501 [sender.py:send_request_log_artifact():1525] logged artifact run-urh96fgp-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM2NjY4NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgyNjY5Nw==', 'latestArtifact': None}} 2025-01-07 20:15:32,551 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:15:32,551 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,551 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,552 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:15:32,552 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now 2025-01-07 20:15:32,552 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:15:32,552 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,555 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,556 DEBUG SenderThread:2221501 [sender.py:send():379] send: telemetry 2025-01-07 20:15:32,556 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,556 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,556 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,557 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,557 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,557 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,557 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,557 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,558 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,558 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,558 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,558 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,558 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,559 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,559 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,559 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,559 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,560 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,560 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,560 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,560 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:32,561 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:32,561 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 20:15:32,806 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/ybong3xl-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 20:15:32,952 INFO wandb-upload_0:2221501 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1yq2keh3 2025-01-07 20:15:32,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:32,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:15:32,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json 2025-01-07 20:15:32,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval 2025-01-07 20:15:33,318 INFO SenderThread:2221501 [sender.py:send_request_log_artifact():1525] logged artifact run-urh96fgp-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM2NjcyMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgyNjcwMg==', 'latestArtifact': None}} 2025-01-07 20:15:33,320 DEBUG SenderThread:2221501 [sender.py:send():379] send: files 2025-01-07 20:15:33,320 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json with policy now 2025-01-07 20:15:33,321 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:15:33,321 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:33,322 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:33,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:33,523 INFO wandb-upload_2:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/yg19ecs8-media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json 2025-01-07 20:15:33,772 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/wcxjbygd-media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json 2025-01-07 20:15:33,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:33,977 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:34,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:35,322 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:35,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:36,272 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 20:15:36,272 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: stop_status 2025-01-07 20:15:36,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:37,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:38,360 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:39,361 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:40,361 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:40,410 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:40,980 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/config.yaml 2025-01-07 20:15:41,361 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:42,361 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:43,285 DEBUG SenderThread:2221501 [sender.py:send():379] send: exit 2025-01-07 20:15:43,285 INFO SenderThread:2221501 [sender.py:send_exit():586] handling exit code: 0 2025-01-07 20:15:43,285 INFO SenderThread:2221501 [sender.py:send_exit():588] handling runtime: 262 2025-01-07 20:15:43,286 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:43,287 INFO SenderThread:2221501 [sender.py:send_exit():594] send defer 2025-01-07 20:15:43,287 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:43,287 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-07 20:15:43,287 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:43,287 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-07 20:15:43,287 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 1 2025-01-07 20:15:43,287 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:43,287 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-07 20:15:43,287 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:43,287 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-07 20:15:43,287 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 2 2025-01-07 20:15:43,287 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:43,287 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-07 20:15:43,288 INFO HandlerThread:2221501 [system_monitor.py:finish():203] Stopping system monitor 2025-01-07 20:15:43,288 INFO HandlerThread:2221501 [interfaces.py:finish():200] Joined cpu monitor 2025-01-07 20:15:43,288 INFO HandlerThread:2221501 [interfaces.py:finish():200] Joined disk monitor 2025-01-07 20:15:43,288 DEBUG SystemMonitor:2221501 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-07 20:15:43,288 DEBUG SystemMonitor:2221501 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-07 20:15:43,341 INFO HandlerThread:2221501 [interfaces.py:finish():200] Joined gpu monitor 2025-01-07 20:15:43,342 INFO HandlerThread:2221501 [interfaces.py:finish():200] Joined memory monitor 2025-01-07 20:15:43,342 INFO HandlerThread:2221501 [interfaces.py:finish():200] Joined network monitor 2025-01-07 20:15:43,342 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:43,342 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-07 20:15:43,342 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 3 2025-01-07 20:15:43,342 DEBUG SenderThread:2221501 [sender.py:send():379] send: stats 2025-01-07 20:15:43,342 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:43,343 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-07 20:15:43,343 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:43,343 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-07 20:15:43,343 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 4 2025-01-07 20:15:43,343 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:43,343 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-07 20:15:43,981 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:43,981 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:49,663 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 20:15:49,664 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:49,664 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: keepalive 2025-01-07 20:15:49,665 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:49,665 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-07 20:15:49,665 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 5 2025-01-07 20:15:49,665 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 20:15:49,665 DEBUG SenderThread:2221501 [sender.py:send():379] send: history 2025-01-07 20:15:49,666 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: summary_record 2025-01-07 20:15:49,666 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:49,666 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-07 20:15:49,667 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:49,667 DEBUG SenderThread:2221501 [sender.py:send():379] send: summary 2025-01-07 20:15:49,667 INFO SenderThread:2221501 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 20:15:49,667 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:49,667 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-07 20:15:49,668 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 6 2025-01-07 20:15:49,668 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:49,668 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-07 20:15:49,668 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:49,668 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-07 20:15:49,668 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 7 2025-01-07 20:15:49,668 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 20:15:49,668 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:49,668 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-07 20:15:49,668 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:49,669 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-07 20:15:49,984 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:50,286 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 20:15:50,457 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 8 2025-01-07 20:15:50,457 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 20:15:50,457 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:50,457 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-07 20:15:50,458 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:50,458 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-07 20:15:50,458 INFO SenderThread:2221501 [job_builder.py:build():440] Attempting to build job artifact 2025-01-07 20:15:50,460 INFO SenderThread:2221501 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-07 20:15:50,462 INFO SenderThread:2221501 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-07 20:15:50,475 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 9 2025-01-07 20:15:50,475 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:50,475 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-07 20:15:50,476 DEBUG SenderThread:2221501 [sender.py:send():379] send: artifact 2025-01-07 20:15:50,781 INFO SenderThread:2221501 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTc1Nzg0Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjE0MDMwNQ==', 'versionIndex': 437}}} 2025-01-07 20:15:50,782 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:50,782 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-07 20:15:50,782 INFO SenderThread:2221501 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-07 20:15:50,984 INFO Thread-12 :2221501 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:50,985 INFO SenderThread:2221501 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files 2025-01-07 20:15:50,985 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/config.yaml config.yaml 2025-01-07 20:15:50,985 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json wandb-summary.json 2025-01-07 20:15:50,985 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:15:50,988 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/requirements.txt requirements.txt 2025-01-07 20:15:50,988 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-metadata.json wandb-metadata.json 2025-01-07 20:15:50,988 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log output.log 2025-01-07 20:15:50,988 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-07 20:15:50,990 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 media/videos/eval/4_eval_4_ff0c295868f146e84066.mp4 2025-01-07 20:15:50,990 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 media/videos/eval/1_eval_1_4296e644b36d9e3925a9.mp4 2025-01-07 20:15:50,990 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 media/videos/eval/3_eval_3_6b814eef54727e0c0f4a.mp4 2025-01-07 20:15:50,990 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 media/videos/eval/2_eval_2_09f9d609000e258b6203.mp4 2025-01-07 20:15:50,990 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 media/videos/eval/0_eval_0_c4ea3df26327a48d1e75.mp4 2025-01-07 20:15:50,993 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json media/table/eval/total_success_rate_ci_6_df810a6369b5406d344d.table.json 2025-01-07 20:15:50,993 INFO SenderThread:2221501 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 20:15:50,993 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 10 2025-01-07 20:15:50,993 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:50,993 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-07 20:15:50,994 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:50,994 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-07 20:15:50,994 INFO SenderThread:2221501 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 20:15:51,160 INFO wandb-upload_2:2221501 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/config.yaml 2025-01-07 20:15:51,164 INFO wandb-upload_1:2221501 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/wandb-summary.json 2025-01-07 20:15:51,210 INFO wandb-upload_3:2221501 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/requirements.txt 2025-01-07 20:15:51,239 INFO wandb-upload_4:2221501 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/files/output.log 2025-01-07 20:15:51,243 INFO wandb-upload_0:2221501 [upload_job.py:push():130] Uploaded file /tmp/tmpq3wiyjefwandb/kcrplpt4-events.out.tfevents.1736298681.leonmkim-ROG-Strix-G15CS-G15CS.2221061.0 2025-01-07 20:15:51,287 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 20:15:51,287 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 20:15:51,443 INFO Thread-11 (_thread_body):2221501 [sender.py:transition_state():614] send defer: 11 2025-01-07 20:15:51,443 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:51,443 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-07 20:15:51,443 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:51,443 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-07 20:15:51,443 INFO SenderThread:2221501 [file_pusher.py:join():175] waiting for file pusher 2025-01-07 20:15:51,445 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 12 2025-01-07 20:15:51,445 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:51,445 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-07 20:15:51,445 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:51,445 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-07 20:15:51,445 INFO SenderThread:2221501 [file_stream.py:finish():601] file stream finish called 2025-01-07 20:15:51,674 INFO SenderThread:2221501 [file_stream.py:finish():605] file stream finish is done 2025-01-07 20:15:51,674 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 13 2025-01-07 20:15:51,674 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:51,674 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-07 20:15:51,674 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:51,674 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-07 20:15:51,674 INFO SenderThread:2221501 [sender.py:transition_state():614] send defer: 14 2025-01-07 20:15:51,674 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: defer 2025-01-07 20:15:51,674 INFO HandlerThread:2221501 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-07 20:15:51,675 DEBUG SenderThread:2221501 [sender.py:send():379] send: final 2025-01-07 20:15:51,675 DEBUG SenderThread:2221501 [sender.py:send():379] send: footer 2025-01-07 20:15:51,675 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: defer 2025-01-07 20:15:51,675 INFO SenderThread:2221501 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-07 20:15:51,675 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 20:15:51,675 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: server_info 2025-01-07 20:15:51,676 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 20:15:51,676 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: get_summary 2025-01-07 20:15:51,676 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 20:15:51,676 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-07 20:15:51,677 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: server_info 2025-01-07 20:15:51,677 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 20:15:51,737 DEBUG SenderThread:2221501 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 20:15:51,737 INFO MainThread:2221501 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-07 20:15:51,738 INFO MainThread:2221501 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-07 20:15:51,738 INFO MainThread:2221501 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-07 20:15:51,738 DEBUG HandlerThread:2221501 [handler.py:handle_request():158] handle_request: shutdown 2025-01-07 20:15:51,738 INFO HandlerThread:2221501 [handler.py:finish():882] shutting down handler 2025-01-07 20:15:52,676 INFO WriterThread:2221501 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/urh96fgp/wandb/run-20250107_201120-urh96fgp/run-urh96fgp.wandb 2025-01-07 20:15:52,737 INFO SenderThread:2221501 [sender.py:finish():1615] shutting down sender 2025-01-07 20:15:52,737 INFO SenderThread:2221501 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 20:15:52,737 INFO SenderThread:2221501 [file_pusher.py:join():175] waiting for file pusher