2025-01-06 18:06:19,073 INFO StreamThr :2098647 [internal.py:wandb_internal():85] W&B internal server running at pid: 2098647, started at: 2025-01-06 18:06:19.072755 2025-01-06 18:06:19,074 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status 2025-01-06 18:06:19,085 INFO WriterThread:2098647 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/run-vb47oelh.wandb 2025-01-06 18:06:19,085 DEBUG SenderThread:2098647 [sender.py:send():379] send: header 2025-01-06 18:06:19,094 DEBUG SenderThread:2098647 [sender.py:send():379] send: run 2025-01-06 18:06:19,421 INFO SenderThread:2098647 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files 2025-01-06 18:06:19,421 INFO SenderThread:2098647 [sender.py:_start_run_threads():1188] run started: vb47oelh with start time 1736204779.072731 2025-01-06 18:06:19,438 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: check_version 2025-01-06 18:06:19,438 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: check_version 2025-01-06 18:06:19,509 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: run_start 2025-01-06 18:06:19,514 DEBUG HandlerThread:2098647 [system_info.py:__init__():26] System info init 2025-01-06 18:06:19,514 DEBUG HandlerThread:2098647 [system_info.py:__init__():41] System info init done 2025-01-06 18:06:19,514 INFO HandlerThread:2098647 [system_monitor.py:start():194] Starting system monitor 2025-01-06 18:06:19,514 INFO SystemMonitor:2098647 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-06 18:06:19,514 INFO HandlerThread:2098647 [system_monitor.py:probe():214] Collecting system info 2025-01-06 18:06:19,515 INFO SystemMonitor:2098647 [interfaces.py:start():188] Started cpu monitoring 2025-01-06 18:06:19,516 INFO SystemMonitor:2098647 [interfaces.py:start():188] Started disk monitoring 2025-01-06 18:06:19,517 INFO SystemMonitor:2098647 [interfaces.py:start():188] Started gpu monitoring 2025-01-06 18:06:19,517 INFO SystemMonitor:2098647 [interfaces.py:start():188] Started memory monitoring 2025-01-06 18:06:19,518 INFO SystemMonitor:2098647 [interfaces.py:start():188] Started network monitoring 2025-01-06 18:06:19,526 DEBUG HandlerThread:2098647 [system_info.py:probe():152] Probing system 2025-01-06 18:06:19,528 DEBUG HandlerThread:2098647 [system_info.py:_probe_git():137] Probing git 2025-01-06 18:06:19,533 DEBUG HandlerThread:2098647 [system_info.py:_probe_git():145] Probing git done 2025-01-06 18:06:19,533 DEBUG HandlerThread:2098647 [system_info.py:probe():200] Probing system done 2025-01-06 18:06:19,533 DEBUG HandlerThread:2098647 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-06T23:06:19.526838', 'startedAt': '2025-01-06T23:06:19.061983', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.11675, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.996, 'min': 800.0, 'max': 4700.0}, {'current': 4500.058, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.002, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 401.7840385437012}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-06 18:06:19,533 INFO HandlerThread:2098647 [system_monitor.py:probe():224] Finished collecting system info 2025-01-06 18:06:19,533 INFO HandlerThread:2098647 [system_monitor.py:probe():227] Publishing system info 2025-01-06 18:06:19,533 DEBUG HandlerThread:2098647 [system_info.py:_save_code():44] Saving code 2025-01-06 18:06:19,534 DEBUG HandlerThread:2098647 [system_info.py:_save_code():67] Saving code done 2025-01-06 18:06:19,534 DEBUG HandlerThread:2098647 [system_info.py:_save_patches():84] Saving git patches 2025-01-06 18:06:19,596 DEBUG HandlerThread:2098647 [system_info.py:_save_patches():126] Saving git patches done 2025-01-06 18:06:19,596 INFO HandlerThread:2098647 [system_monitor.py:probe():229] Finished publishing system info 2025-01-06 18:06:19,599 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:06:19,599 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-06 18:06:19,600 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-06 18:06:19,600 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-06 18:06:19,838 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: python_packages 2025-01-06 18:06:19,838 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: python_packages 2025-01-06 18:06:19,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:06:19,839 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:06:19,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:19,842 INFO HandlerThread:2098647 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/tb" save: true } control { relay_id: "4213779cbc0c4ea28496f8328555bbcc" } _info { stream_id: "vb47oelh" } 2025-01-06 18:06:19,842 DEBUG HandlerThread:2098647 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-06 18:06:19,849 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/28oyl0y4-wandb-metadata.json 2025-01-06 18:06:19,876 INFO wandb-upload_1:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/3qzd1pu5-code/FISH/eval_robot.py 2025-01-06 18:06:19,887 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/pgurgypv-diff.patch 2025-01-06 18:06:19,996 DEBUG SenderThread:2098647 [sender.py:send():379] send: telemetry 2025-01-06 18:06:19,996 DEBUG SenderThread:2098647 [sender.py:send():379] send: tbrecord 2025-01-06 18:06:19,997 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:06:19,997 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 with policy live 2025-01-06 18:06:20,182 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/qm58b42e-events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/requirements.txt 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/diff.patch 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/code/FISH/eval_robot.py 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-metadata.json 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:06:20,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/code/FISH 2025-01-06 18:06:20,424 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/code 2025-01-06 18:06:20,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:21,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:22,422 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:22,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:23,669 DEBUG SenderThread:2098647 [sender.py:send():379] send: config 2025-01-06 18:06:23,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:24,677 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:24,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:25,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:26,423 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:26,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:27,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:28,424 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:28,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:29,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:29,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:30,424 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:30,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:31,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:32,425 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:06:32,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:33,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:34,838 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:06:34,839 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:06:34,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:34,995 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:35,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:36,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:37,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:38,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:39,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:40,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:40,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:41,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:42,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:43,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:44,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:45,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:46,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:46,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:47,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:48,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:49,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:06:49,839 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:06:49,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:50,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:51,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:51,929 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:52,430 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/config.yaml 2025-01-06 18:06:52,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:53,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:54,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:55,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:56,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:57,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:06:57,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:58,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:06:59,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:00,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:01,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:02,433 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:02,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:03,471 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:03,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:04,434 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:04,839 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:07:04,839 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:07:04,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:05,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:06,434 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:06,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:07,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:08,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:08,928 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:09,847 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:10,847 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:11,847 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:12,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:13,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:14,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:14,849 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:15,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:16,849 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:17,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:18,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:19,518 DEBUG SystemMonitor:2098647 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-06 18:07:19,519 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:07:19,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:07:19,841 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:07:19,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:19,918 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:20,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:21,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:22,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:23,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:24,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:25,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:25,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:26,850 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:27,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:28,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:29,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:30,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:30,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:31,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:32,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:33,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:34,840 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:07:34,841 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:07:34,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:35,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:36,151 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:36,853 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:37,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:38,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:39,469 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:07:39,469 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 with policy now 2025-01-06 18:07:39,471 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:07:39,472 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:07:39,472 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:07:39,473 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:07:39,719 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/k9t2peu7-media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 2025-01-06 18:07:39,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:40,443 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:40,443 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 2025-01-06 18:07:40,443 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:07:40,444 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval 2025-01-06 18:07:40,444 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos 2025-01-06 18:07:40,444 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media 2025-01-06 18:07:40,718 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/khbhrbwo-media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 2025-01-06 18:07:40,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:41,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:41,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:42,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:43,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:44,445 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:44,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:45,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:46,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:47,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:47,858 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:48,446 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:48,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:49,519 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:07:49,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:07:49,841 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:07:49,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:50,864 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:51,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:52,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:52,995 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:53,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:54,855 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:55,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:56,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:57,449 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:07:57,701 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/fboc6tt8-events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:07:57,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:58,449 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:07:58,677 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:07:58,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:07:59,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:00,450 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:00,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:01,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:02,451 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:02,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:03,856 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:04,327 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:04,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:08:04,842 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:08:04,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:05,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:06,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:07,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:08,858 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:09,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:09,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:10,859 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:11,858 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:12,858 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:13,864 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:14,454 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:14,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:14,859 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:15,860 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:16,860 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:17,860 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:18,861 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:19,519 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:08:19,841 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:08:19,842 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:08:19,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:19,983 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:20,861 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:21,862 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:22,457 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:22,861 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:23,862 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:24,862 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:25,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:25,865 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:26,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:27,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:28,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:29,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:30,459 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:30,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:30,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:31,864 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:32,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:33,863 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:34,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:08:34,843 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:08:34,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:35,869 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:36,349 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:36,461 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:36,618 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:08:36,618 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 with policy now 2025-01-06 18:08:36,623 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:08:36,624 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:08:36,624 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:08:36,729 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:08:36,845 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/di80x26n-media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 2025-01-06 18:08:36,873 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:37,461 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:08:37,462 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 2025-01-06 18:08:37,462 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval 2025-01-06 18:08:37,729 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/5v4ah568-media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 2025-01-06 18:08:37,876 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:38,462 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:38,870 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:39,875 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:40,871 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:41,871 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:42,313 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:42,463 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:42,875 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:43,874 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:44,464 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:44,873 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:45,879 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:46,880 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:47,848 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:47,880 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:48,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:49,466 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:08:49,520 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:08:49,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:08:49,843 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:08:49,878 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:50,309 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:08:50,309 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:08:50,309 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:08:50,466 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:50,467 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:08:50,879 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:51,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:52,467 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:52,882 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:53,166 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:53,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:54,468 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:54,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:55,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:56,469 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:08:56,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:57,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:58,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:08:58,934 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:08:59,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:00,881 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:01,882 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:02,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:03,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:04,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:09:04,842 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:09:04,889 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:04,980 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:05,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:06,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:07,884 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:08,883 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:09,884 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:10,848 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:10,884 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:11,884 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:12,884 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:13,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:14,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:15,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:16,848 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:16,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:17,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:18,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:19,520 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:09:19,842 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:09:19,842 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:09:19,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:20,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:21,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:21,973 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:22,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:23,885 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:24,886 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:25,886 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:26,886 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:27,848 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:27,886 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:28,886 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:29,895 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:30,480 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:30,626 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:09:30,626 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 with policy now 2025-01-06 18:09:30,637 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:09:30,638 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:09:30,638 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:09:30,660 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:09:30,856 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/ircl3c9e-media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 2025-01-06 18:09:30,900 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:31,481 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:09:31,481 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 2025-01-06 18:09:31,481 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval 2025-01-06 18:09:31,746 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/8t37ch96-media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 2025-01-06 18:09:31,896 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:32,481 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:32,902 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:33,675 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:33,897 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:34,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:09:34,851 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:09:34,897 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:35,910 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:36,482 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:36,902 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:37,901 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:38,483 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:38,899 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:39,249 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:39,908 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:40,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:41,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:42,906 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:43,151 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:09:43,151 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:09:43,152 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:09:43,486 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:09:43,486 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:09:43,736 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/c73vqglr-events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:09:43,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:44,487 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:44,705 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:44,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:45,916 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:46,908 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:47,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:48,489 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:48,908 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:49,521 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:09:49,844 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:09:49,844 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:09:49,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:49,940 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:50,490 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:09:50,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:51,909 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:52,910 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:53,910 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:54,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:55,853 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:09:55,911 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:56,911 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:57,911 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:58,911 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:09:59,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:00,853 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:00,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:01,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:02,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:03,912 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:04,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:10:04,843 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:10:04,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:05,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:05,994 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:06,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:07,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:08,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:09,913 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:10,914 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:11,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:11,914 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:12,914 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:13,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:14,914 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:15,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:16,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:17,854 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:17,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:18,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:19,521 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:10:19,843 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:10:19,843 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:10:19,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:20,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:21,915 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:22,500 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:22,921 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:23,406 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:23,695 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:10:23,695 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 with policy now 2025-01-06 18:10:23,697 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:10:23,698 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:10:23,698 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:10:23,714 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:10:23,923 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:23,931 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/lcary7gx-media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 2025-01-06 18:10:24,501 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:24,501 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:10:24,501 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 2025-01-06 18:10:24,502 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval 2025-01-06 18:10:24,764 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/7ysestke-media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 2025-01-06 18:10:24,924 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:25,935 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:26,923 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:27,926 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:28,437 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:28,502 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:28,923 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:29,928 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:30,926 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:31,927 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:32,504 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:32,926 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:33,933 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:34,233 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:34,851 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:10:34,851 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:10:34,936 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:35,929 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:36,930 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:37,945 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:38,938 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:39,861 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:39,936 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:40,933 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:41,223 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:10:41,223 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:10:41,224 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:10:41,507 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:10:41,507 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:10:41,945 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:42,507 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:42,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:43,945 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:44,508 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:44,945 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:45,858 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:45,954 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:46,509 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:46,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:47,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:48,509 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:10:48,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:49,522 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:10:49,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:10:49,845 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:10:49,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:50,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:50,996 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:51,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:52,946 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:53,947 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:54,947 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:55,947 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:56,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:10:56,947 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:57,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:58,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:10:59,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:00,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:01,857 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:01,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:02,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:03,948 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:04,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:11:04,845 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:11:04,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:05,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:06,927 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:06,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:07,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:08,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:09,949 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:10,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:11,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:12,859 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:12,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:13,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:14,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:15,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:16,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:17,950 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:18,859 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:18,951 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:19,523 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:11:19,845 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:11:19,845 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:11:19,951 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:20,951 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:21,951 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:22,957 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:23,212 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:11:23,212 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 with policy now 2025-01-06 18:11:23,216 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:11:23,216 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:11:23,217 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:23,218 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:23,439 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/3oo7pnjy-media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 2025-01-06 18:11:23,523 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:23,523 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 2025-01-06 18:11:23,523 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval 2025-01-06 18:11:23,787 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/yc2x18el-media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 2025-01-06 18:11:23,967 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:24,231 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:24,523 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:24,955 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:25,979 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:26,969 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:27,963 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:28,524 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:28,966 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:29,688 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:29,962 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:30,525 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:30,959 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:31,967 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:32,526 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:11:32,766 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/j460tiwh-events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:11:32,964 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:33,287 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:11:33,287 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:33,287 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:33,351 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 18:11:33,351 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 18:11:33,526 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:33,526 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-06 18:11:33,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media 2025-01-06 18:11:33,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval 2025-01-06 18:11:33,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table 2025-01-06 18:11:33,540 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:11:33,697 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/qfa78be5-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-06 18:11:33,845 INFO wandb-upload_2:2098647 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp6qmyzqmn 2025-01-06 18:11:33,923 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 18:11:33,931 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 18:11:33,956 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:34,227 INFO SenderThread:2098647 [sender.py:send_request_log_artifact():1525] logged artifact run-vb47oelh-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTk5MDU4NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxODI0OA==', 'latestArtifact': None}} 2025-01-06 18:11:34,227 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:11:34,227 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now 2025-01-06 18:11:34,227 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:11:34,227 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,228 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,228 DEBUG SenderThread:2098647 [sender.py:send():379] send: telemetry 2025-01-06 18:11:34,228 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,232 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,232 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,232 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,232 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,233 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,233 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,233 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,233 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,234 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,234 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,234 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,235 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,235 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,235 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,235 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,236 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,236 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,236 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,236 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,236 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:34,237 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:34,237 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 18:11:34,467 INFO wandb-upload_1:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/o8lfi9aq-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-06 18:11:34,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:34,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:34,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:11:34,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json 2025-01-06 18:11:34,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval 2025-01-06 18:11:34,649 INFO wandb-upload_2:2098647 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpqycxkxgi 2025-01-06 18:11:34,789 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/ye57dor5-media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json 2025-01-06 18:11:34,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:11:34,956 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:35,024 INFO SenderThread:2098647 [sender.py:send_request_log_artifact():1525] logged artifact run-vb47oelh-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTk5MDU5OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxODI0OQ==', 'latestArtifact': None}} 2025-01-06 18:11:35,024 DEBUG SenderThread:2098647 [sender.py:send():379] send: files 2025-01-06 18:11:35,025 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:35,025 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json with policy now 2025-01-06 18:11:35,025 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:11:35,025 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:35,026 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:35,026 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:11:35,271 INFO wandb-upload_1:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/f9qg76k0-media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json 2025-01-06 18:11:35,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:35,956 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:36,527 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:36,956 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:37,957 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:38,957 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:39,957 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:40,873 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:40,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:41,957 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:42,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:43,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:44,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:45,873 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:45,960 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:46,960 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:47,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:48,959 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:49,523 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:11:49,846 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 18:11:49,846 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: stop_status 2025-01-06 18:11:49,958 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:11:50,914 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:51,184 DEBUG SenderThread:2098647 [sender.py:send():379] send: exit 2025-01-06 18:11:51,184 INFO SenderThread:2098647 [sender.py:send_exit():586] handling exit code: 0 2025-01-06 18:11:51,184 INFO SenderThread:2098647 [sender.py:send_exit():588] handling runtime: 331 2025-01-06 18:11:51,184 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:51,184 INFO SenderThread:2098647 [sender.py:send_exit():594] send defer 2025-01-06 18:11:51,184 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:51,185 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-06 18:11:51,185 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:51,185 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-06 18:11:51,185 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 1 2025-01-06 18:11:51,185 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:51,185 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-06 18:11:51,185 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:51,185 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-06 18:11:51,185 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 2 2025-01-06 18:11:51,185 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:51,186 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-06 18:11:51,186 INFO HandlerThread:2098647 [system_monitor.py:finish():203] Stopping system monitor 2025-01-06 18:11:51,186 DEBUG SystemMonitor:2098647 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-06 18:11:51,186 DEBUG SystemMonitor:2098647 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-06 18:11:51,186 INFO HandlerThread:2098647 [interfaces.py:finish():200] Joined cpu monitor 2025-01-06 18:11:51,187 INFO HandlerThread:2098647 [interfaces.py:finish():200] Joined disk monitor 2025-01-06 18:11:51,221 INFO HandlerThread:2098647 [interfaces.py:finish():200] Joined gpu monitor 2025-01-06 18:11:51,221 INFO HandlerThread:2098647 [interfaces.py:finish():200] Joined memory monitor 2025-01-06 18:11:51,221 INFO HandlerThread:2098647 [interfaces.py:finish():200] Joined network monitor 2025-01-06 18:11:51,222 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:51,222 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-06 18:11:51,222 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 3 2025-01-06 18:11:51,222 DEBUG SenderThread:2098647 [sender.py:send():379] send: stats 2025-01-06 18:11:51,222 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:51,222 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-06 18:11:51,222 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:51,222 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-06 18:11:51,222 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 4 2025-01-06 18:11:51,223 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:51,223 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-06 18:11:51,534 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/config.yaml 2025-01-06 18:11:51,534 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:52,535 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:57,292 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:11:57,292 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:57,292 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:57,292 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-06 18:11:57,292 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 5 2025-01-06 18:11:57,292 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: keepalive 2025-01-06 18:11:57,292 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:11:57,293 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:57,294 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-06 18:11:57,293 DEBUG SenderThread:2098647 [sender.py:send():379] send: history 2025-01-06 18:11:57,294 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: summary_record 2025-01-06 18:11:57,294 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:57,295 DEBUG SenderThread:2098647 [sender.py:send():379] send: summary 2025-01-06 18:11:57,298 INFO SenderThread:2098647 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 18:11:57,299 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:57,299 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-06 18:11:57,299 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 6 2025-01-06 18:11:57,299 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:57,299 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-06 18:11:57,299 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:57,299 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-06 18:11:57,299 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 7 2025-01-06 18:11:57,299 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 18:11:57,299 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:57,299 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-06 18:11:57,300 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:57,300 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-06 18:11:57,537 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:57,916 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:11:58,349 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 8 2025-01-06 18:11:58,349 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:11:58,349 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:58,349 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-06 18:11:58,350 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:58,350 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-06 18:11:58,350 INFO SenderThread:2098647 [job_builder.py:build():440] Attempting to build job artifact 2025-01-06 18:11:58,353 INFO SenderThread:2098647 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-06 18:11:58,355 INFO SenderThread:2098647 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-06 18:11:58,370 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 9 2025-01-06 18:11:58,370 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:58,370 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-06 18:11:58,370 DEBUG SenderThread:2098647 [sender.py:send():379] send: artifact 2025-01-06 18:11:58,537 INFO Thread-12 :2098647 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:58,810 INFO wandb-upload_2:2098647 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1mspmxl3 2025-01-06 18:11:58,811 INFO wandb-upload_1:2098647 [upload_job.py:push():85] Skipped uploading /tmp/tmprros7r4x/wandb-job.json 2025-01-06 18:11:58,916 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:11:58,939 INFO wandb-upload_0:2098647 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpntr5kgdh 2025-01-06 18:11:59,520 INFO SenderThread:2098647 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk3NDE3OA==', 'versionIndex': 426}}} 2025-01-06 18:11:59,520 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:59,520 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-06 18:11:59,520 INFO SenderThread:2098647 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-06 18:11:59,538 INFO SenderThread:2098647 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files 2025-01-06 18:11:59,538 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/config.yaml config.yaml 2025-01-06 18:11:59,538 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json wandb-summary.json 2025-01-06 18:11:59,539 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:11:59,539 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/requirements.txt requirements.txt 2025-01-06 18:11:59,539 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-metadata.json wandb-metadata.json 2025-01-06 18:11:59,539 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log output.log 2025-01-06 18:11:59,542 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/diff.patch diff.patch 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 media/videos/eval/2_eval_2_71bebcde765f8093e4db.mp4 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 media/videos/eval/3_eval_3_9667046e025c1a04f584.mp4 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 media/videos/eval/0_eval_0_a21f553da3e495971337.mp4 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 media/videos/eval/4_eval_4_537861e0f91853c73080.mp4 2025-01-06 18:11:59,543 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 media/videos/eval/1_eval_1_bbe5d527784bc078331e.mp4 2025-01-06 18:11:59,546 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json media/table/eval/total_success_rate_ci_6_e32f206babc2825ce217.table.json 2025-01-06 18:11:59,546 INFO SenderThread:2098647 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-06 18:11:59,546 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 10 2025-01-06 18:11:59,546 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:11:59,548 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:59,548 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-06 18:11:59,548 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:59,548 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-06 18:11:59,548 INFO SenderThread:2098647 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 18:11:59,727 INFO wandb-upload_1:2098647 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/wandb-summary.json 2025-01-06 18:11:59,752 INFO wandb-upload_2:2098647 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/config.yaml 2025-01-06 18:11:59,765 INFO wandb-upload_3:2098647 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/requirements.txt 2025-01-06 18:11:59,780 INFO wandb-upload_4:2098647 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/files/output.log 2025-01-06 18:11:59,785 INFO wandb-upload_0:2098647 [upload_job.py:push():130] Uploaded file /tmp/tmpgjte8_b0wandb/mmn2rth2-events.out.tfevents.1736204779.leonmkim-ROG-Strix-G15CS-G15CS.2098452.0 2025-01-06 18:11:59,917 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:11:59,917 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:11:59,985 INFO Thread-11 (_thread_body):2098647 [sender.py:transition_state():614] send defer: 11 2025-01-06 18:11:59,985 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:59,985 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-06 18:11:59,985 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:59,985 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-06 18:11:59,985 INFO SenderThread:2098647 [file_pusher.py:join():175] waiting for file pusher 2025-01-06 18:11:59,987 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 12 2025-01-06 18:11:59,987 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:11:59,987 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-06 18:11:59,987 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:11:59,987 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-06 18:11:59,987 INFO SenderThread:2098647 [file_stream.py:finish():601] file stream finish called 2025-01-06 18:12:00,178 INFO SenderThread:2098647 [file_stream.py:finish():605] file stream finish is done 2025-01-06 18:12:00,178 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 13 2025-01-06 18:12:00,178 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:12:00,178 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-06 18:12:00,178 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:12:00,178 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-06 18:12:00,179 INFO SenderThread:2098647 [sender.py:transition_state():614] send defer: 14 2025-01-06 18:12:00,179 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: defer 2025-01-06 18:12:00,179 INFO HandlerThread:2098647 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-06 18:12:00,179 DEBUG SenderThread:2098647 [sender.py:send():379] send: final 2025-01-06 18:12:00,179 DEBUG SenderThread:2098647 [sender.py:send():379] send: footer 2025-01-06 18:12:00,179 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: defer 2025-01-06 18:12:00,179 INFO SenderThread:2098647 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-06 18:12:00,180 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:12:00,180 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:12:00,180 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: server_info 2025-01-06 18:12:00,180 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: server_info 2025-01-06 18:12:00,183 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 18:12:00,183 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: get_summary 2025-01-06 18:12:00,184 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-06 18:12:00,184 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 18:12:00,232 DEBUG SenderThread:2098647 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 18:12:00,232 INFO MainThread:2098647 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-06 18:12:00,232 INFO MainThread:2098647 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-06 18:12:00,233 INFO MainThread:2098647 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-06 18:12:00,233 DEBUG HandlerThread:2098647 [handler.py:handle_request():158] handle_request: shutdown 2025-01-06 18:12:00,233 INFO HandlerThread:2098647 [handler.py:finish():882] shutting down handler 2025-01-06 18:12:01,184 INFO WriterThread:2098647 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/vb47oelh/wandb/run-20250106_180619-vb47oelh/run-vb47oelh.wandb 2025-01-06 18:12:01,232 INFO SenderThread:2098647 [sender.py:finish():1615] shutting down sender 2025-01-06 18:12:01,232 INFO SenderThread:2098647 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 18:12:01,232 INFO SenderThread:2098647 [file_pusher.py:join():175] waiting for file pusher