2025-01-07 20:05:09,486 INFO    StreamThr :2220142 [internal.py:wandb_internal():85] W&B internal server running at pid: 2220142, started at: 2025-01-07 20:05:09.486108
2025-01-07 20:05:09,487 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status
2025-01-07 20:05:09,501 INFO    WriterThread:2220142 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/run-8gt3cc6e.wandb
2025-01-07 20:05:09,501 DEBUG   SenderThread:2220142 [sender.py:send():379] send: header
2025-01-07 20:05:09,509 DEBUG   SenderThread:2220142 [sender.py:send():379] send: run
2025-01-07 20:05:09,796 INFO    SenderThread:2220142 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files
2025-01-07 20:05:09,796 INFO    SenderThread:2220142 [sender.py:_start_run_threads():1188] run started: 8gt3cc6e with start time 1736298309.486238
2025-01-07 20:05:09,812 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: check_version
2025-01-07 20:05:09,812 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: check_version
2025-01-07 20:05:09,887 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: run_start
2025-01-07 20:05:09,892 DEBUG   HandlerThread:2220142 [system_info.py:__init__():26] System info init
2025-01-07 20:05:09,892 DEBUG   HandlerThread:2220142 [system_info.py:__init__():41] System info init done
2025-01-07 20:05:09,892 INFO    HandlerThread:2220142 [system_monitor.py:start():194] Starting system monitor
2025-01-07 20:05:09,892 INFO    SystemMonitor:2220142 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-07 20:05:09,892 INFO    HandlerThread:2220142 [system_monitor.py:probe():214] Collecting system info
2025-01-07 20:05:09,892 INFO    SystemMonitor:2220142 [interfaces.py:start():188] Started cpu monitoring
2025-01-07 20:05:09,893 INFO    SystemMonitor:2220142 [interfaces.py:start():188] Started disk monitoring
2025-01-07 20:05:09,893 INFO    SystemMonitor:2220142 [interfaces.py:start():188] Started gpu monitoring
2025-01-07 20:05:09,894 INFO    SystemMonitor:2220142 [interfaces.py:start():188] Started memory monitoring
2025-01-07 20:05:09,894 INFO    SystemMonitor:2220142 [interfaces.py:start():188] Started network monitoring
2025-01-07 20:05:09,906 DEBUG   HandlerThread:2220142 [system_info.py:probe():152] Probing system
2025-01-07 20:05:09,907 DEBUG   HandlerThread:2220142 [system_info.py:_probe_git():137] Probing git
2025-01-07 20:05:09,912 DEBUG   HandlerThread:2220142 [system_info.py:_probe_git():145] Probing git done
2025-01-07 20:05:09,912 DEBUG   HandlerThread:2220142 [system_info.py:probe():200] Probing system done
2025-01-07 20:05:09,912 DEBUG   HandlerThread:2220142 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T01:05:09.906073', 'startedAt': '2025-01-08T01:05:09.471590', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.9895, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.951, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 541.9574737548828}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-07 20:05:09,912 INFO    HandlerThread:2220142 [system_monitor.py:probe():224] Finished collecting system info
2025-01-07 20:05:09,912 INFO    HandlerThread:2220142 [system_monitor.py:probe():227] Publishing system info
2025-01-07 20:05:09,912 DEBUG   HandlerThread:2220142 [system_info.py:_save_code():44] Saving code
2025-01-07 20:05:09,914 DEBUG   HandlerThread:2220142 [system_info.py:_save_code():67] Saving code done
2025-01-07 20:05:09,914 DEBUG   HandlerThread:2220142 [system_info.py:_save_patches():84] Saving git patches
2025-01-07 20:05:09,948 DEBUG   HandlerThread:2220142 [system_info.py:_save_patches():126] Saving git patches done
2025-01-07 20:05:09,949 INFO    HandlerThread:2220142 [system_monitor.py:probe():229] Finished publishing system info
2025-01-07 20:05:09,952 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:05:09,952 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-07 20:05:09,952 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-07 20:05:10,196 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: python_packages
2025-01-07 20:05:10,196 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: python_packages
2025-01-07 20:05:10,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:05:10,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:05:10,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:10,199 INFO    HandlerThread:2220142 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/tb"
  save: true
}
control {
  relay_id: "62165ca8d82b422782b83b5601ae078f"
}
_info {
  stream_id: "8gt3cc6e"
}

2025-01-07 20:05:10,199 DEBUG   HandlerThread:2220142 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-07 20:05:10,203 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/e7l7e743-wandb-metadata.json
2025-01-07 20:05:10,265 DEBUG   SenderThread:2220142 [sender.py:send():379] send: telemetry
2025-01-07 20:05:10,265 DEBUG   SenderThread:2220142 [sender.py:send():379] send: tbrecord
2025-01-07 20:05:10,266 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:05:10,266 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0 with policy live
2025-01-07 20:05:10,283 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/j9xmsrem-code/FISH/eval_robot.py
2025-01-07 20:05:10,451 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/295elqwg-events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:05:10,798 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:10,798 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-metadata.json
2025-01-07 20:05:10,798 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/code/FISH/eval_robot.py
2025-01-07 20:05:10,798 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/requirements.txt
2025-01-07 20:05:10,799 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:05:10,799 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/code
2025-01-07 20:05:10,799 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/code/FISH
2025-01-07 20:05:11,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:12,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:12,797 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:13,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:14,002 DEBUG   SenderThread:2220142 [sender.py:send():379] send: config
2025-01-07 20:05:14,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:15,010 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:15,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:16,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:16,798 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:17,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:18,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:18,799 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:19,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:20,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:20,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:20,800 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:21,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:22,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:22,800 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:23,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:24,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:24,801 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:25,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:05:25,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:05:25,237 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:25,316 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:26,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:26,802 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:27,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:28,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:28,802 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:29,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:30,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:31,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:31,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:32,200 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:32,803 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:05:33,200 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:34,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:35,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:36,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:37,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:37,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:38,203 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:39,203 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:40,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:05:40,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:05:40,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:41,203 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:42,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:42,269 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:42,806 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/config.yaml
2025-01-07 20:05:43,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:44,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:45,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:46,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:47,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:48,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:48,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:49,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:50,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:51,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:52,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:53,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:54,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:05:54,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:55,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:05:55,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:05:55,245 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:56,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:57,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:58,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:59,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:05:59,271 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:00,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:01,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:02,206 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:02,698 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:06:02,698 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4 with policy now
2025-01-07 20:06:02,699 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:06:02,700 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:06:02,700 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:06:02,700 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:06:02,811 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:02,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4
2025-01-07 20:06:02,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:06:02,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media
2025-01-07 20:06:02,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval
2025-01-07 20:06:02,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos
2025-01-07 20:06:02,967 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/d2czae8a-media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4
2025-01-07 20:06:03,206 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:03,241 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/fwfcssii-media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4
2025-01-07 20:06:04,207 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:04,710 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:04,812 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:05,207 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:06,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:07,207 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:08,207 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:08,813 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:09,207 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:09,894 DEBUG   SystemMonitor:2220142 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-07 20:06:09,895 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:06:09,895 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:10,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:06:10,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:06:10,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:10,814 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:11,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:12,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:13,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:14,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:15,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:15,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:16,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:17,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:18,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:19,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:20,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:20,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:21,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:22,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:23,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:24,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:25,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:06:25,197 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:06:25,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:25,276 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:26,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:27,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:28,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:29,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:30,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:31,199 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:31,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:32,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:33,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:33,822 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:06:34,078 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/8yxk2hzl-events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:06:34,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:34,822 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:35,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:36,200 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:36,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:36,823 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:37,213 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:38,213 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:38,824 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:39,213 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:39,896 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:06:40,197 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:06:40,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:06:40,214 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:40,825 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:41,214 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:41,258 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:42,217 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:43,216 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:44,217 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:44,826 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:06:45,216 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:46,216 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:47,200 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:47,217 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:48,217 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:49,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:50,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:51,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:52,203 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:52,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:53,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:54,218 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:55,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:06:55,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:06:55,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:56,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:57,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:57,278 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:06:58,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:06:59,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:00,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:01,221 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:02,221 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:02,317 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:03,221 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:04,221 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:04,832 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:05,224 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:06,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:07,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:08,200 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:08,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:09,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:09,896 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:07:10,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:07:10,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:07:10,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:11,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:12,223 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:13,224 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:13,307 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:13,353 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:07:13,354 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:07:13,354 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4 with policy now
2025-01-07 20:07:13,354 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:07:13,354 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:07:13,354 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:07:13,631 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/jk6b8tia-media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4
2025-01-07 20:07:13,835 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:07:13,835 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4
2025-01-07 20:07:13,835 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval
2025-01-07 20:07:14,110 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/vrhbh0sd-media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4
2025-01-07 20:07:14,226 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:14,835 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:15,230 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:16,227 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:17,232 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:18,229 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:18,837 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:18,985 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:19,232 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:20,229 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:20,837 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:21,230 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:22,230 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:23,230 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:24,201 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:24,230 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:25,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:07:25,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:07:25,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:26,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:27,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:28,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:28,640 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:07:28,640 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:07:28,641 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:07:28,840 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:07:28,840 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:07:29,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:29,245 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:30,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:30,841 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:31,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:32,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:32,842 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:33,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:34,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:34,842 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:07:35,153 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:35,231 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:36,232 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:37,232 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:38,232 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:39,233 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:39,897 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:07:40,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:07:40,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:07:40,245 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:40,267 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:41,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:42,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:43,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:44,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:45,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:46,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:46,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:47,234 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:48,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:49,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:50,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:51,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:51,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:52,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:53,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:54,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:55,198 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:07:55,198 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:07:55,241 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:56,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:56,282 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:07:57,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:58,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:07:59,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:00,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:00,852 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:01,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:02,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:02,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:03,237 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:04,236 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:05,237 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:06,237 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:07,238 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:07,620 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:08,180 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:08:08,180 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4 with policy now
2025-01-07 20:08:08,183 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:08:08,183 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:08:08,183 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:08:08,184 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:08:08,242 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:08,448 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/00boltkl-media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4
2025-01-07 20:08:08,855 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:08,855 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:08:08,855 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4
2025-01-07 20:08:08,856 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval
2025-01-07 20:08:09,165 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/r5tnco6y-media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4
2025-01-07 20:08:09,243 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:09,897 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:08:10,203 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:08:10,203 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:08:10,249 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:10,856 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:11,245 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:12,245 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:13,085 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:13,244 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:14,244 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:14,857 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:15,243 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:16,248 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:16,858 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:17,246 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:18,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:18,246 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:19,249 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:20,254 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:21,252 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:22,249 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:23,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:23,252 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:24,248 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:25,201 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:08:25,201 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:08:25,253 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:26,253 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:27,253 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:28,256 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:28,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:29,257 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:30,255 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:31,255 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:32,255 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:33,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:34,206 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:34,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:35,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:36,263 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:37,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:38,262 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:39,242 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:39,259 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:39,504 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:08:39,504 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:08:39,505 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:08:39,865 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:08:39,866 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:08:39,897 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:08:40,151 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/9fcc669c-events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:08:40,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:08:40,204 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:08:40,260 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:40,866 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:41,267 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:42,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:42,866 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:43,260 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:44,260 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:44,867 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:08:45,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:45,260 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:46,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:47,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:48,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:49,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:50,208 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:50,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:51,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:52,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:53,261 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:54,262 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:55,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:08:55,202 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:08:55,262 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:55,364 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:08:56,262 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:57,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:58,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:08:59,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:00,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:01,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:01,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:02,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:03,264 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:04,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:05,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:06,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:07,209 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:07,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:08,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:09,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:09,898 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:09:10,202 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:09:10,202 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:09:10,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:11,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:12,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:12,353 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:13,265 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:14,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:15,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:16,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:17,266 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:17,920 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:18,276 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:18,878 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:19,226 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:09:19,226 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_b35454858e15f45db555.mp4 with policy now
2025-01-07 20:09:19,235 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:09:19,236 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:09:19,236 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:09:19,243 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:09:19,270 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:19,501 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/6lyono38-media/videos/eval/3_eval_3_b35454858e15f45db555.mp4
2025-01-07 20:09:19,879 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:09:19,879 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/3_eval_3_b35454858e15f45db555.mp4
2025-01-07 20:09:19,880 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval
2025-01-07 20:09:20,174 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/xts3snmt-media/videos/eval/3_eval_3_b35454858e15f45db555.mp4
2025-01-07 20:09:20,285 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:20,880 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:21,281 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:22,281 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:22,974 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:23,280 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:24,280 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:24,881 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:25,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:09:25,205 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:09:25,280 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:26,279 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:26,882 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:27,282 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:28,283 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:28,817 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:29,285 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:30,288 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:31,284 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:32,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:33,285 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:34,215 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:34,286 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:35,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:36,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:37,288 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:37,885 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:09:38,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:38,578 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:09:38,578 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:09:38,578 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:09:38,885 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:38,886 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:09:39,293 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:39,805 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:39,898 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:09:40,205 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:09:40,205 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:09:40,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:40,886 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:41,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:42,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:42,887 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:43,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:44,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:45,213 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:45,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:46,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:47,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:48,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:49,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:50,214 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:50,289 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:51,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:52,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:53,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:54,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:55,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:09:55,204 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:09:55,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:55,333 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:09:56,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:57,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:58,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:09:58,892 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:09:59,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:00,290 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:00,472 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:01,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:02,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:02,894 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:03,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:04,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:05,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:06,214 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:06,291 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:06,895 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:07,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:08,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:09,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:09,899 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:10:10,204 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:10:10,204 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:10:10,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:11,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:11,310 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:12,292 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:13,293 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:14,293 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:15,293 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:16,293 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:16,313 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:17,300 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:17,614 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:10:17,614 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4 with policy now
2025-01-07 20:10:17,616 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:10:17,617 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:10:17,617 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:17,627 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:17,882 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/0c7ap176-media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4
2025-01-07 20:10:17,899 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:17,899 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4
2025-01-07 20:10:17,899 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval
2025-01-07 20:10:18,191 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/08idz8if-media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4
2025-01-07 20:10:18,311 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:18,899 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:19,302 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:20,308 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:21,309 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:21,381 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:22,304 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:22,900 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:23,310 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:24,314 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:25,210 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:10:25,210 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:10:25,309 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:26,317 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:26,902 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:27,312 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:27,369 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:28,311 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:29,314 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:30,325 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:31,329 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:32,326 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:33,227 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:33,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:34,327 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:35,324 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:36,345 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:36,648 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:10:36,648 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:36,649 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:36,756 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 20:10:36,756 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 20:10:36,905 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:10:36,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:36,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json
2025-01-07 20:10:36,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media
2025-01-07 20:10:36,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table
2025-01-07 20:10:36,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval
2025-01-07 20:10:36,950 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:10:37,145 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/u8grp3lf-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json
2025-01-07 20:10:37,170 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/lk3h76u6-events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:10:37,266 INFO    wandb-upload_2:2220142 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpzcs05d44
2025-01-07 20:10:37,321 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:37,375 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 20:10:37,421 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 20:10:37,705 INFO    SenderThread:2220142 [sender.py:send_request_log_artifact():1525] logged artifact run-8gt3cc6e-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM1ODYzNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgyNDY0MQ==', 'latestArtifact': None}}
2025-01-07 20:10:37,705 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:10:37,705 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now
2025-01-07 20:10:37,705 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:10:37,705 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,706 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,706 DEBUG   SenderThread:2220142 [sender.py:send():379] send: telemetry
2025-01-07 20:10:37,706 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,710 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,710 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,710 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,710 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,711 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,711 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,711 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,711 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,712 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,712 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,712 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,712 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,713 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,713 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,713 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,713 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,714 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,714 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,714 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,714 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:37,715 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:37,715 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 20:10:37,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:10:37,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:37,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json
2025-01-07 20:10:37,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval
2025-01-07 20:10:37,969 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/8j0kqxm8-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json
2025-01-07 20:10:38,088 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/4n1dp5b1-media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json
2025-01-07 20:10:38,093 INFO    wandb-upload_2:2220142 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppc617cxg
2025-01-07 20:10:38,321 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:38,538 INFO    SenderThread:2220142 [sender.py:send_request_log_artifact():1525] logged artifact run-8gt3cc6e-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM1ODY1OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgyNDY0NQ==', 'latestArtifact': None}}
2025-01-07 20:10:38,539 DEBUG   SenderThread:2220142 [sender.py:send():379] send: files
2025-01-07 20:10:38,539 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json with policy now
2025-01-07 20:10:38,539 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:10:38,539 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:38,540 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:38,540 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:38,830 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/2wg74seq-media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json
2025-01-07 20:10:38,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:38,906 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:39,321 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:39,899 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:10:40,206 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 20:10:40,206 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: stop_status
2025-01-07 20:10:40,321 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:41,321 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:42,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:43,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:44,305 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:44,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:45,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:46,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:47,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:48,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:49,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:50,220 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:50,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:51,322 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:52,323 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:10:52,437 DEBUG   SenderThread:2220142 [sender.py:send():379] send: exit
2025-01-07 20:10:52,438 INFO    SenderThread:2220142 [sender.py:send_exit():586] handling exit code: 0
2025-01-07 20:10:52,438 INFO    SenderThread:2220142 [sender.py:send_exit():588] handling runtime: 342
2025-01-07 20:10:52,438 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:52,438 INFO    SenderThread:2220142 [sender.py:send_exit():594] send defer
2025-01-07 20:10:52,438 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:52,438 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-07 20:10:52,439 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:52,439 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-07 20:10:52,439 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 1
2025-01-07 20:10:52,439 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:52,439 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-07 20:10:52,439 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:52,439 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-07 20:10:52,439 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 2
2025-01-07 20:10:52,439 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:52,439 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-07 20:10:52,439 INFO    HandlerThread:2220142 [system_monitor.py:finish():203] Stopping system monitor
2025-01-07 20:10:52,439 INFO    HandlerThread:2220142 [interfaces.py:finish():200] Joined cpu monitor
2025-01-07 20:10:52,440 INFO    HandlerThread:2220142 [interfaces.py:finish():200] Joined disk monitor
2025-01-07 20:10:52,440 DEBUG   SystemMonitor:2220142 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-07 20:10:52,440 DEBUG   SystemMonitor:2220142 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-07 20:10:52,494 INFO    HandlerThread:2220142 [interfaces.py:finish():200] Joined gpu monitor
2025-01-07 20:10:52,494 INFO    HandlerThread:2220142 [interfaces.py:finish():200] Joined memory monitor
2025-01-07 20:10:52,494 INFO    HandlerThread:2220142 [interfaces.py:finish():200] Joined network monitor
2025-01-07 20:10:52,494 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:52,494 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-07 20:10:52,494 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 3
2025-01-07 20:10:52,494 DEBUG   SenderThread:2220142 [sender.py:send():379] send: stats
2025-01-07 20:10:52,495 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:52,495 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-07 20:10:52,495 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:52,495 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-07 20:10:52,495 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 4
2025-01-07 20:10:52,495 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:52,495 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-07 20:10:52,911 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:54,912 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:10:58,656 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:10:58,656 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:58,656 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:58,656 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-07 20:10:58,656 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: keepalive
2025-01-07 20:10:58,657 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 5
2025-01-07 20:10:58,658 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:10:58,658 DEBUG   SenderThread:2220142 [sender.py:send():379] send: history
2025-01-07 20:10:58,658 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: summary_record
2025-01-07 20:10:58,658 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:58,658 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-07 20:10:58,659 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:58,659 DEBUG   SenderThread:2220142 [sender.py:send():379] send: summary
2025-01-07 20:10:58,663 INFO    SenderThread:2220142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 20:10:58,663 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:58,663 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-07 20:10:58,663 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 6
2025-01-07 20:10:58,663 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:58,663 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-07 20:10:58,663 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:58,663 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-07 20:10:58,667 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 20:10:58,807 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 7
2025-01-07 20:10:58,807 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:10:58,807 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-07 20:10:58,807 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:10:58,807 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-07 20:10:58,914 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/config.yaml
2025-01-07 20:10:58,914 INFO    Thread-12 :2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:10:59,439 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:11:02,357 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 8
2025-01-07 20:11:02,357 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:11:02,357 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:02,358 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-07 20:11:02,358 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:02,358 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-07 20:11:02,358 INFO    SenderThread:2220142 [job_builder.py:build():440] Attempting to build job artifact
2025-01-07 20:11:02,360 INFO    SenderThread:2220142 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-07 20:11:02,362 INFO    SenderThread:2220142 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-07 20:11:02,374 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 9
2025-01-07 20:11:02,374 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:02,374 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-07 20:11:02,374 DEBUG   SenderThread:2220142 [sender.py:send():379] send: artifact
2025-01-07 20:11:02,440 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:11:02,558 INFO    SenderThread:2220142 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTc1Nzg0Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjE0MDMwNQ==', 'versionIndex': 437}}}
2025-01-07 20:11:02,559 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:02,559 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-07 20:11:02,559 INFO    SenderThread:2220142 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-07 20:11:02,915 INFO    SenderThread:2220142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:11:02,915 INFO    SenderThread:2220142 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files
2025-01-07 20:11:02,915 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/config.yaml config.yaml
2025-01-07 20:11:02,916 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json wandb-summary.json
2025-01-07 20:11:02,916 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/requirements.txt requirements.txt
2025-01-07 20:11:02,916 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-metadata.json wandb-metadata.json
2025-01-07 20:11:02,916 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0 events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:11:02,916 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log output.log
2025-01-07 20:11:02,919 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-07 20:11:02,921 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4 media/videos/eval/1_eval_1_4e6b8d6b47dcf1f2852e.mp4
2025-01-07 20:11:02,921 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/3_eval_3_b35454858e15f45db555.mp4 media/videos/eval/3_eval_3_b35454858e15f45db555.mp4
2025-01-07 20:11:02,921 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4 media/videos/eval/2_eval_2_f300fd7a4cb461e89098.mp4
2025-01-07 20:11:02,922 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4 media/videos/eval/0_eval_0_94f52a23cf863a3c1137.mp4
2025-01-07 20:11:02,922 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4 media/videos/eval/4_eval_4_93d560b9b8a81ed37bb5.mp4
2025-01-07 20:11:02,923 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json media/table/eval/total_success_rate_ci_6_130e835d6612d43dcccd.table.json
2025-01-07 20:11:02,923 INFO    SenderThread:2220142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json
2025-01-07 20:11:02,923 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 10
2025-01-07 20:11:02,924 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:11:02,925 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:02,925 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-07 20:11:02,925 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:02,925 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-07 20:11:02,925 INFO    SenderThread:2220142 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 20:11:03,108 INFO    wandb-upload_0:2220142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/wandb-summary.json
2025-01-07 20:11:03,117 INFO    wandb-upload_2:2220142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/config.yaml
2025-01-07 20:11:03,119 INFO    wandb-upload_4:2220142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/output.log
2025-01-07 20:11:03,126 INFO    wandb-upload_1:2220142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/files/requirements.txt
2025-01-07 20:11:03,224 INFO    wandb-upload_3:2220142 [upload_job.py:push():130] Uploaded file /tmp/tmplx5lrfugwandb/oc0glxge-events.out.tfevents.1736298310.leonmkim-ROG-Strix-G15CS-G15CS.2219968.0
2025-01-07 20:11:03,424 INFO    Thread-11 (_thread_body):2220142 [sender.py:transition_state():614] send defer: 11
2025-01-07 20:11:03,424 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:03,424 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-07 20:11:03,424 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:03,425 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-07 20:11:03,425 INFO    SenderThread:2220142 [file_pusher.py:join():175] waiting for file pusher
2025-01-07 20:11:03,426 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 12
2025-01-07 20:11:03,426 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:03,426 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-07 20:11:03,426 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:03,426 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-07 20:11:03,426 INFO    SenderThread:2220142 [file_stream.py:finish():601] file stream finish called
2025-01-07 20:11:03,440 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:11:03,649 INFO    SenderThread:2220142 [file_stream.py:finish():605] file stream finish is done
2025-01-07 20:11:03,650 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 13
2025-01-07 20:11:03,650 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:11:03,650 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:03,650 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-07 20:11:03,650 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:03,650 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-07 20:11:03,650 INFO    SenderThread:2220142 [sender.py:transition_state():614] send defer: 14
2025-01-07 20:11:03,650 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: defer
2025-01-07 20:11:03,650 INFO    HandlerThread:2220142 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-07 20:11:03,651 DEBUG   SenderThread:2220142 [sender.py:send():379] send: final
2025-01-07 20:11:03,651 DEBUG   SenderThread:2220142 [sender.py:send():379] send: footer
2025-01-07 20:11:03,651 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: defer
2025-01-07 20:11:03,651 INFO    SenderThread:2220142 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-07 20:11:03,651 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:11:03,651 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: server_info
2025-01-07 20:11:03,652 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 20:11:03,652 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:11:03,652 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: get_summary
2025-01-07 20:11:03,652 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: server_info
2025-01-07 20:11:03,652 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-07 20:11:03,654 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 20:11:03,703 DEBUG   SenderThread:2220142 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 20:11:03,704 INFO    MainThread:2220142 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-07 20:11:03,704 INFO    MainThread:2220142 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-07 20:11:03,704 INFO    MainThread:2220142 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-07 20:11:03,704 DEBUG   HandlerThread:2220142 [handler.py:handle_request():158] handle_request: shutdown
2025-01-07 20:11:03,705 INFO    HandlerThread:2220142 [handler.py:finish():882] shutting down handler
2025-01-07 20:11:04,652 INFO    WriterThread:2220142 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/8gt3cc6e/wandb/run-20250107_200509-8gt3cc6e/run-8gt3cc6e.wandb
2025-01-07 20:11:04,704 INFO    SenderThread:2220142 [sender.py:finish():1615] shutting down sender
2025-01-07 20:11:04,704 INFO    SenderThread:2220142 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 20:11:04,704 INFO    SenderThread:2220142 [file_pusher.py:join():175] waiting for file pusher