2025-01-07 22:06:10,457 INFO    StreamThr :2240208 [internal.py:wandb_internal():85] W&B internal server running at pid: 2240208, started at: 2025-01-07 22:06:10.456569
2025-01-07 22:06:10,458 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status
2025-01-07 22:06:10,472 INFO    WriterThread:2240208 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/run-4fmihp69.wandb
2025-01-07 22:06:10,473 DEBUG   SenderThread:2240208 [sender.py:send():379] send: header
2025-01-07 22:06:10,481 DEBUG   SenderThread:2240208 [sender.py:send():379] send: run
2025-01-07 22:06:10,744 INFO    SenderThread:2240208 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files
2025-01-07 22:06:10,744 INFO    SenderThread:2240208 [sender.py:_start_run_threads():1188] run started: 4fmihp69 with start time 1736305570.457261
2025-01-07 22:06:10,762 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: check_version
2025-01-07 22:06:10,763 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: check_version
2025-01-07 22:06:10,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: run_start
2025-01-07 22:06:10,813 DEBUG   HandlerThread:2240208 [system_info.py:__init__():26] System info init
2025-01-07 22:06:10,813 DEBUG   HandlerThread:2240208 [system_info.py:__init__():41] System info init done
2025-01-07 22:06:10,813 INFO    HandlerThread:2240208 [system_monitor.py:start():194] Starting system monitor
2025-01-07 22:06:10,813 INFO    SystemMonitor:2240208 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-07 22:06:10,813 INFO    SystemMonitor:2240208 [interfaces.py:start():188] Started cpu monitoring
2025-01-07 22:06:10,814 INFO    HandlerThread:2240208 [system_monitor.py:probe():214] Collecting system info
2025-01-07 22:06:10,814 INFO    SystemMonitor:2240208 [interfaces.py:start():188] Started disk monitoring
2025-01-07 22:06:10,814 INFO    SystemMonitor:2240208 [interfaces.py:start():188] Started gpu monitoring
2025-01-07 22:06:10,815 INFO    SystemMonitor:2240208 [interfaces.py:start():188] Started memory monitoring
2025-01-07 22:06:10,815 INFO    SystemMonitor:2240208 [interfaces.py:start():188] Started network monitoring
2025-01-07 22:06:10,826 DEBUG   HandlerThread:2240208 [system_info.py:probe():152] Probing system
2025-01-07 22:06:10,828 DEBUG   HandlerThread:2240208 [system_info.py:_probe_git():137] Probing git
2025-01-07 22:06:10,832 DEBUG   HandlerThread:2240208 [system_info.py:_probe_git():145] Probing git done
2025-01-07 22:06:10,832 DEBUG   HandlerThread:2240208 [system_info.py:probe():200] Probing system done
2025-01-07 22:06:10,832 DEBUG   HandlerThread:2240208 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T03:06:10.826762', 'startedAt': '2025-01-08T03:06:10.446482', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3937.0912500000004, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.994, 'min': 800.0, 'max': 4700.0}, {'current': 4499.94, 'min': 800.0, 'max': 4700.0}, {'current': 4497.561, 'min': 800.0, 'max': 4700.0}, {'current': 4500.022, 'min': 800.0, 'max': 4700.0}, {'current': 4498.542, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 657.744571685791}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-07 22:06:10,832 INFO    HandlerThread:2240208 [system_monitor.py:probe():224] Finished collecting system info
2025-01-07 22:06:10,832 INFO    HandlerThread:2240208 [system_monitor.py:probe():227] Publishing system info
2025-01-07 22:06:10,832 DEBUG   HandlerThread:2240208 [system_info.py:_save_code():44] Saving code
2025-01-07 22:06:10,834 DEBUG   HandlerThread:2240208 [system_info.py:_save_code():67] Saving code done
2025-01-07 22:06:10,834 DEBUG   HandlerThread:2240208 [system_info.py:_save_patches():84] Saving git patches
2025-01-07 22:06:10,896 DEBUG   HandlerThread:2240208 [system_info.py:_save_patches():126] Saving git patches done
2025-01-07 22:06:10,896 INFO    HandlerThread:2240208 [system_monitor.py:probe():229] Finished publishing system info
2025-01-07 22:06:10,900 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:06:10,900 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-07 22:06:10,900 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-07 22:06:10,900 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-07 22:06:11,143 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/qqyp17un-diff.patch
2025-01-07 22:06:11,144 INFO    wandb-upload_0:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/0vvacken-wandb-metadata.json
2025-01-07 22:06:11,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: python_packages
2025-01-07 22:06:11,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:06:11,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: python_packages
2025-01-07 22:06:11,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:11,145 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:06:11,147 INFO    HandlerThread:2240208 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/tb"
  save: true
}
control {
  relay_id: "eac7231c22d04ed2b4484c3efc83ccb3"
}
_info {
  stream_id: "4fmihp69"
}

2025-01-07 22:06:11,147 DEBUG   HandlerThread:2240208 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-07 22:06:11,172 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ei73ugkb-code/FISH/eval_robot.py
2025-01-07 22:06:11,267 DEBUG   SenderThread:2240208 [sender.py:send():379] send: telemetry
2025-01-07 22:06:11,267 DEBUG   SenderThread:2240208 [sender.py:send():379] send: tbrecord
2025-01-07 22:06:11,268 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:06:11,268 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0 with policy live
2025-01-07 22:06:11,446 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/wk7l1bwy-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/requirements.txt
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/code/FISH/eval_robot.py
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/diff.patch
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:06:11,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-metadata.json
2025-01-07 22:06:11,747 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/code/FISH
2025-01-07 22:06:11,747 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/code
2025-01-07 22:06:12,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:13,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:13,745 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:14,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:15,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:16,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:16,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:16,468 DEBUG   SenderThread:2240208 [sender.py:send():379] send: config
2025-01-07 22:06:17,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:17,746 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:18,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:19,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:19,747 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:20,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:21,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:21,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:21,748 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:22,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:23,146 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:23,748 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:24,146 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:25,146 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:25,749 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:26,143 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:06:26,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:06:26,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:26,275 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:27,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:28,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:29,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:30,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:31,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:32,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:32,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:33,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:33,751 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:34,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:35,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:35,752 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:36,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:37,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:37,589 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:37,752 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:38,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:39,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:40,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:41,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:06:41,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:06:41,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:42,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:43,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:43,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:43,754 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/config.yaml
2025-01-07 22:06:44,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:45,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:46,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:47,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:48,150 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:49,150 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:49,249 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:49,755 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:50,151 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:51,151 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:52,151 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:53,151 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:53,756 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:54,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:54,457 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:06:55,158 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:55,757 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:06:56,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:06:56,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:06:56,189 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:57,158 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:58,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:06:59,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:00,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:00,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:01,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:02,160 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:03,160 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:04,160 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:04,381 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:07:04,382 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:07:04,382 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4 with policy now
2025-01-07 22:07:04,382 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:07:04,382 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:07:04,383 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:07:04,605 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/etwnhs8s-media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4
2025-01-07 22:07:04,759 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:07:04,760 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4
2025-01-07 22:07:04,760 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media
2025-01-07 22:07:04,760 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:07:04,760 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos
2025-01-07 22:07:05,034 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/nwn9bqku-media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4
2025-01-07 22:07:05,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:05,384 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:05,760 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:06,165 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:07,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:08,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:09,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:09,761 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:10,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:10,469 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:10,815 DEBUG   SystemMonitor:2240208 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-07 22:07:10,816 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:07:11,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:07:11,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:07:11,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:11,762 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:12,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:13,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:14,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:15,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:16,153 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:16,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:17,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:18,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:19,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:20,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:20,765 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:07:21,017 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/6ib9bsap-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:07:21,154 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:21,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:21,765 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:22,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:23,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:23,766 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:24,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:25,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:25,767 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:26,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:07:26,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:07:26,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:26,256 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:27,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:28,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:29,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:30,169 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:31,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:32,157 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:32,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:33,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:34,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:35,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:36,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:37,158 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:37,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:38,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:39,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:40,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:40,817 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:07:41,144 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:07:41,144 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:07:41,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:42,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:42,238 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:43,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:44,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:45,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:46,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:47,174 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:48,155 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:48,174 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:49,175 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:49,774 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:50,174 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:51,175 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:52,177 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:52,277 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:07:52,277 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:07:52,278 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4 with policy now
2025-01-07 22:07:52,278 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:07:52,278 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:07:52,278 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:07:52,554 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/pjwmo6o9-media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4
2025-01-07 22:07:52,775 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:07:52,776 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4
2025-01-07 22:07:52,776 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:07:53,049 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/4mryielv-media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4
2025-01-07 22:07:53,177 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:53,279 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:53,776 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:54,180 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:55,180 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:56,147 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:07:56,148 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:07:56,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:57,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:57,777 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:07:58,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:58,413 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:07:59,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:07:59,778 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:08:00,180 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:01,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:02,180 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:03,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:04,157 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:04,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:05,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:06,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:07,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:08,181 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:09,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:10,156 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:10,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:10,817 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:08:11,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:08:11,145 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:08:11,189 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:12,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:13,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:14,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:15,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:15,259 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:16,183 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:17,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:18,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:19,186 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:20,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:21,157 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:21,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:22,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:23,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:23,786 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:08:24,033 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/17nqx6vr-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:08:24,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:24,586 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:08:24,587 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:08:24,588 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:08:24,786 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:08:25,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:25,786 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:08:26,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:08:26,146 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:08:26,189 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:26,307 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:27,186 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:28,186 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:29,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:29,788 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:08:30,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:31,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:32,157 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:32,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:33,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:34,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:35,187 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:36,188 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:37,158 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:37,188 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:38,188 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:39,188 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:40,188 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:40,818 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:08:41,145 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:08:41,146 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:08:41,189 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:42,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:42,385 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:43,189 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:44,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:45,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:46,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:47,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:48,158 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:48,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:49,190 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:50,191 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:51,191 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:52,191 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:53,159 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:53,191 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:54,191 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:55,193 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:55,729 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:08:55,729 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4 with policy now
2025-01-07 22:08:55,730 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:08:55,731 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:08:55,731 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:08:55,731 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:08:55,798 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:08:55,798 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:08:55,799 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4
2025-01-07 22:08:55,799 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:08:55,956 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/5w232wyh-media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4
2025-01-07 22:08:56,152 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:08:56,153 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:08:56,192 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:56,219 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/qm16amsm-media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4
2025-01-07 22:08:57,194 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:57,799 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:08:58,197 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:08:58,218 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:08:59,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:00,196 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:01,197 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:01,800 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:02,196 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:03,200 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:03,801 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:04,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:04,206 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:05,204 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:05,412 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:09:05,413 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:09:05,413 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:09:05,802 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:09:05,802 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:09:06,206 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:07,206 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:07,802 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:08,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:09,192 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:09,233 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:10,206 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:10,818 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:09:11,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:09:11,148 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:09:11,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:11,804 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:12,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:13,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:14,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:14,306 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:15,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:16,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:17,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:18,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:19,207 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:20,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:20,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:21,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:22,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:23,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:24,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:25,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:25,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:26,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:09:26,148 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:09:26,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:27,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:28,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:29,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:30,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:30,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:31,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:32,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:33,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:34,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:35,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:36,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:36,209 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:37,211 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:37,767 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:09:37,767 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4 with policy now
2025-01-07 22:09:37,769 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:09:37,770 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:09:37,770 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:09:37,770 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:09:37,813 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:09:37,813 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:37,813 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4
2025-01-07 22:09:37,813 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:09:37,997 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/nul1chui-media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4
2025-01-07 22:09:38,213 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:38,256 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/811s36x6-media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4
2025-01-07 22:09:39,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:39,813 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:40,216 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:40,819 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:09:41,152 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:09:41,152 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:09:41,223 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:41,305 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:42,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:43,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:43,815 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:44,214 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:45,216 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:45,815 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:46,216 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:47,166 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:47,218 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:48,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:49,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:49,467 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:09:49,467 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:09:49,468 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:09:49,817 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:09:49,817 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:09:50,067 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/k0239dli-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:09:50,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:51,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:51,817 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:52,167 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:52,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:53,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:53,818 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:54,219 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:55,219 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:55,819 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:09:56,148 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:09:56,149 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:09:56,219 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:57,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:57,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:09:58,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:09:59,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:00,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:01,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:02,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:03,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:03,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:04,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:05,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:06,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:07,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:08,163 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:08,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:09,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:10,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:10,820 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:10:11,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:10:11,149 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:10:11,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:12,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:13,223 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:13,248 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:14,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:15,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:16,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:17,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:18,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:19,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:19,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:20,224 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:21,237 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:22,219 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:10:22,219 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4 with policy now
2025-01-07 22:10:22,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:10:22,221 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:10:22,221 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:10:22,222 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:10:22,237 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:22,463 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/a01kcc8d-media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4
2025-01-07 22:10:22,829 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:10:22,829 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4
2025-01-07 22:10:22,829 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:10:23,132 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/4sm2g1pq-media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4
2025-01-07 22:10:23,227 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:23,829 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:10:24,235 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:24,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:25,234 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:26,155 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:10:26,155 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:10:26,243 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:27,244 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:27,831 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:10:28,253 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:29,266 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:29,565 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:29,831 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:10:30,275 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:31,267 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:32,261 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:33,260 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:34,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:35,167 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:35,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:36,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:37,269 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:37,834 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:10:38,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:38,530 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:10:38,530 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:10:38,531 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:10:38,834 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:10:39,269 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:39,835 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:10:40,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:40,820 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:10:40,820 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:41,155 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:10:41,155 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:10:41,275 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:42,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:43,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:43,836 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:10:44,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:45,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:45,841 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:46,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:47,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:48,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:49,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:50,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:51,167 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:51,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:52,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:53,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:54,271 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:55,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:56,151 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:10:56,152 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:10:56,223 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:10:56,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:57,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:58,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:10:59,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:00,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:01,272 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:02,167 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:02,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:03,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:04,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:05,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:06,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:07,274 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:07,625 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:07,845 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:08,274 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:09,278 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:09,827 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:11:09,827 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4 with policy now
2025-01-07 22:11:09,828 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:11:09,829 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:11:09,829 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:11:09,830 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:11:09,845 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:11:09,846 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:09,846 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4
2025-01-07 22:11:09,846 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:11:10,047 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/wu3rai74-media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4
2025-01-07 22:11:10,274 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:10,296 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/e4l5q7rx-media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4
2025-01-07 22:11:10,820 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:11:11,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:11:11,161 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:11:11,282 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:11,846 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:12,296 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:13,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:13,329 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:14,283 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:15,290 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:15,847 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:16,279 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:17,284 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:17,848 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:18,286 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:19,169 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:19,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:20,300 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:21,304 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:22,296 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:22,584 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:11:22,584 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:11:22,584 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:11:22,850 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:11:22,850 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:11:23,111 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/yk35ljkk-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:11:23,301 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:23,850 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:24,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:24,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:25,300 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:25,851 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:26,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:11:26,161 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:11:26,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:27,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:27,852 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:28,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:29,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:29,694 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:30,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:31,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:32,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:33,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:34,297 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:35,173 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:35,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:36,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:37,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:38,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:39,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:40,175 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:40,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:40,821 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:11:41,155 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:11:41,155 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:11:41,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:42,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:43,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:44,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:45,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:45,312 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:46,298 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:47,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:48,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:49,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:50,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:50,869 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:51,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:51,861 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:52,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:53,299 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:54,309 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:54,981 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:11:54,981 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4 with policy now
2025-01-07 22:11:54,991 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:11:54,992 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:11:54,992 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:11:54,992 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:11:55,195 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/wncg1hnt-media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4
2025-01-07 22:11:55,302 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:55,862 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:11:55,863 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:11:55,863 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4
2025-01-07 22:11:55,863 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:11:56,022 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:11:56,148 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/zdp0t2t3-media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4
2025-01-07 22:11:56,176 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:11:56,176 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:11:56,305 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:57,309 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:58,315 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:59,318 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:11:59,864 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:00,314 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:01,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:01,310 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:02,314 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:03,318 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:03,865 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:04,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:05,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:06,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:06,799 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:06,866 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:12:07,333 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:07,640 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:12:07,640 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:12:07,641 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:12:07,867 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:12:07,867 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:12:07,867 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:08,326 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:09,326 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:09,867 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:10,328 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:10,821 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:12:11,157 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:12:11,157 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:12:11,329 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:11,868 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:12,217 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:12,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:13,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:13,869 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:14,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:15,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:16,330 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:17,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:18,173 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:18,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:19,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:20,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:21,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:22,331 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:23,173 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:23,332 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:24,339 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:25,339 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:25,873 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:26,155 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:12:26,155 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:12:26,339 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:27,339 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:28,273 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:28,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:29,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:30,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:31,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:32,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:33,340 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:34,173 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:34,341 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:35,342 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:36,342 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:37,342 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:38,342 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:39,174 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:39,342 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:40,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:40,822 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:12:41,162 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:12:41,162 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4 with policy now
2025-01-07 22:12:41,164 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:12:41,164 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:12:41,174 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:12:41,237 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:12:41,237 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:12:41,237 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:12:41,349 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:41,413 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/hzfb5ngp-media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4
2025-01-07 22:12:41,879 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:12:41,879 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:41,880 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4
2025-01-07 22:12:41,880 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:12:42,153 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/qhxnxdkp-media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4
2025-01-07 22:12:42,362 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:42,880 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:43,375 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:44,238 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:44,365 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:45,371 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:45,881 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:46,369 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:47,374 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:48,372 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:49,384 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:49,498 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:49,882 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:50,372 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:51,371 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:52,380 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:53,382 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:54,375 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:55,196 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:12:55,377 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:56,172 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:12:56,173 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:12:56,384 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:56,700 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:12:56,700 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:12:56,701 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:12:56,885 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:12:56,885 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:12:57,113 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/yjvxbtxq-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:12:57,390 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:57,885 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:12:58,390 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:59,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:12:59,886 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:00,417 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:00,532 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:01,398 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:01,887 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:02,400 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:03,392 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:03,888 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:04,392 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:05,392 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:05,541 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:05,889 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:06,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:07,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:08,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:09,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:09,890 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:10,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:10,822 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:13:10,823 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:11,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:13:11,163 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:13:11,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:12,393 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:13,394 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:14,394 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:15,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:15,909 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:16,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:17,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:17,894 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:18,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:19,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:20,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:21,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:21,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:22,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:23,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:24,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:25,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:26,162 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:13:26,163 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:13:26,261 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:26,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:27,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:28,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:29,396 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:30,404 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:30,851 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:13:30,851 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4 with policy now
2025-01-07 22:13:30,855 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:13:30,855 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:13:30,855 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:13:30,855 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:13:30,900 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:13:30,900 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4
2025-01-07 22:13:30,901 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:13:31,081 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/d26k68mv-media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4
2025-01-07 22:13:31,350 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/9crv7elw-media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4
2025-01-07 22:13:31,428 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:31,900 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:32,179 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:32,426 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:33,404 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:34,408 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:35,403 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:35,902 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:36,414 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:37,403 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:37,903 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:38,093 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:38,427 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:39,426 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:40,409 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:40,823 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:13:41,175 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:13:41,176 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:13:41,430 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:42,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:43,319 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:43,411 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:43,905 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:13:44,415 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:44,762 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:13:44,762 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:13:44,762 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:13:44,905 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:13:45,426 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:45,906 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:46,425 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:47,418 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:48,426 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:48,817 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:49,427 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:49,907 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:50,424 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:51,424 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:52,420 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:53,420 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:53,909 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:13:54,030 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:54,420 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:55,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:56,165 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:13:56,165 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:13:56,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:57,429 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:58,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:59,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:13:59,430 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:13:59,911 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:00,421 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:01,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:01,912 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:02,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:03,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:04,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:05,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:05,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:06,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:07,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:08,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:09,422 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:10,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:10,823 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:14:10,823 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:11,165 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:14:11,165 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:14:11,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:12,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:13,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:14,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:15,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:16,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:16,423 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:17,424 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:18,424 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:19,426 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:19,919 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:19,951 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:14:19,951 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_c7576684fb563a572588.mp4 with policy now
2025-01-07 22:14:19,955 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:14:19,955 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:14:19,955 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:14:19,956 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:14:20,170 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/jz8ps62b-media/videos/eval/9_eval_9_c7576684fb563a572588.mp4
2025-01-07 22:14:20,444 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:20,919 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:14:20,919 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/9_eval_9_c7576684fb563a572588.mp4
2025-01-07 22:14:20,920 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:14:21,200 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:21,202 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/rzw69i32-media/videos/eval/9_eval_9_c7576684fb563a572588.mp4
2025-01-07 22:14:21,439 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:21,919 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:22,457 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:23,443 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:24,450 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:25,459 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:25,921 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:26,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:14:26,185 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:14:26,315 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:26,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:27,456 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:27,922 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:28,452 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:29,462 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:30,455 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:31,452 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:32,204 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:32,459 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:32,820 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:14:32,821 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:14:32,821 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:14:32,924 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:14:32,924 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:14:33,165 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/6elnrjsu-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:14:33,489 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:33,924 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:34,470 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:35,463 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:35,925 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:36,479 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:37,474 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:37,913 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:37,926 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:38,483 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:39,480 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:40,475 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:40,824 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:14:41,185 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:14:41,185 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:14:41,479 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:42,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:43,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:43,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:43,928 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:44,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:45,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:46,464 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:47,465 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:48,194 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:48,465 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:49,465 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:50,465 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:51,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:52,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:53,466 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:53,933 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:14:54,101 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:54,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:55,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:56,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:14:56,170 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:14:56,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:57,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:58,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:14:59,305 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:14:59,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:00,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:01,467 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:02,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:03,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:04,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:05,197 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:05,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:06,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:07,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:08,468 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:09,493 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:09,941 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:10,029 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:15:10,029 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4 with policy now
2025-01-07 22:15:10,069 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:15:10,069 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:15:10,070 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:15:10,070 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:15:10,287 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ct0xpxq0-media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4
2025-01-07 22:15:10,476 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:10,825 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:15:10,827 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:10,941 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:15:10,942 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4
2025-01-07 22:15:10,942 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:15:11,213 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:15:11,214 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:15:11,227 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/cft7widw-media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4
2025-01-07 22:15:11,507 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:11,942 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:12,493 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:12,942 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:13,497 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:14,487 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:15,486 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:15,943 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:16,225 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:16,498 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:17,507 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:17,944 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:18,493 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:19,496 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:20,497 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:21,493 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:22,095 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:22,495 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:22,886 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:15:22,886 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:15:22,886 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:15:22,946 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:15:22,946 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:15:23,502 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:23,946 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:24,525 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:25,516 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:26,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:15:26,221 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:15:26,500 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:27,306 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:27,520 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:27,948 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:28,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:29,518 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:30,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:31,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:31,949 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:32,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:33,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:33,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:34,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:35,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:36,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:37,508 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:38,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:38,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:39,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:40,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:40,825 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:15:41,193 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:15:41,193 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:15:41,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:42,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:43,315 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:43,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:44,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:45,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:46,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:47,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:47,956 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:48,377 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:48,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:49,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:49,956 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:50,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:51,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:52,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:53,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:54,220 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:54,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:55,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:56,193 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:15:56,194 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:15:56,510 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:57,512 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:57,960 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:15:58,515 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:58,605 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:15:58,605 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4 with policy now
2025-01-07 22:15:58,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:15:58,613 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:15:58,613 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:15:58,614 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:15:58,818 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/1f2moux3-media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4
2025-01-07 22:15:58,960 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:15:58,960 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4
2025-01-07 22:15:58,960 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:15:59,226 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/8tz12dd2-media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4
2025-01-07 22:15:59,518 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:15:59,674 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:15:59,960 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:00,524 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:01,523 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:02,541 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:03,538 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:03,962 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:04,522 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:04,937 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:05,540 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:05,963 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:06,543 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:07,541 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:08,553 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:09,542 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:10,246 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:10,546 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:10,826 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:16:11,199 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:16:11,199 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:16:11,546 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:11,945 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:16:11,945 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:16:11,946 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:16:11,965 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:16:11,965 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:16:12,547 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:12,966 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:16:13,565 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:13,966 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:14,552 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:15,579 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:16,020 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:16,566 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:17,568 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:17,968 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:18,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:19,574 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:20,574 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:21,243 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:21,576 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:22,567 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:23,567 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:23,970 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:24,568 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:25,568 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:26,196 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:16:26,196 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:16:26,290 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:26,568 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:27,569 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:28,569 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:29,569 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:30,569 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:31,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:31,602 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:32,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:33,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:33,974 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:34,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:35,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:36,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:37,234 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:37,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:37,976 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:38,570 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:39,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:40,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:40,826 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:16:41,196 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:16:41,196 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:16:41,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:42,253 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:42,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:43,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:44,572 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:45,571 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:46,573 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:47,572 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:47,980 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:48,234 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:48,573 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:49,592 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:49,981 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:50,432 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:16:50,432 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4 with policy now
2025-01-07 22:16:50,438 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:16:50,439 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:16:50,439 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:16:50,439 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:16:50,580 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:50,655 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/2iqgnveh-media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4
2025-01-07 22:16:50,981 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:16:50,981 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4
2025-01-07 22:16:50,982 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:16:51,269 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/lhqe1hin-media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4
2025-01-07 22:16:51,594 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:51,982 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:52,589 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:52,982 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:53,254 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:53,602 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:54,585 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:55,608 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:55,983 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:56,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:16:56,223 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:16:56,612 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:57,590 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:57,984 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:16:58,598 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:16:59,249 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:16:59,596 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:00,605 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:01,619 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:01,986 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:17:01,986 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:02,010 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:17:02,010 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:17:02,010 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:17:02,236 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/eh9nqv6k-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:17:02,618 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:02,986 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:17:03,619 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:03,986 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:04,632 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:04,793 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:05,633 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:06,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:07,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:07,989 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:08,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:09,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:09,917 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:09,990 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:10,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:10,827 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:17:11,201 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:17:11,201 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:17:11,610 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:12,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:13,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:14,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:15,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:15,918 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:15,993 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:16,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:17,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:18,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:19,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:20,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:21,251 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:21,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:22,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:23,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:23,996 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:24,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:25,613 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:26,201 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:17:26,201 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:17:26,303 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:26,614 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:27,614 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:27,997 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:28,614 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:29,614 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:29,998 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:30,624 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:31,624 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:31,999 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:32,252 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:32,625 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:33,638 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:34,000 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:34,145 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:17:34,145 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4 with policy now
2025-01-07 22:17:34,156 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:17:34,156 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:17:34,156 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:17:34,156 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:17:34,372 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/s26jid66-media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4
2025-01-07 22:17:34,635 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:35,001 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:17:35,001 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4
2025-01-07 22:17:35,001 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:17:35,310 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/s9dcdqos-media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4
2025-01-07 22:17:35,654 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:36,001 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:36,666 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:37,002 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:37,256 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:37,647 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:38,647 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:39,649 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:40,003 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:40,653 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:40,828 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:17:41,201 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:17:41,201 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:17:41,664 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:42,004 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:42,348 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:42,660 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:43,664 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:44,665 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:45,663 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:46,676 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:47,663 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:47,926 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:48,006 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:17:48,064 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:17:48,064 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:17:48,064 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:17:48,670 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:49,006 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:17:49,007 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:17:49,659 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:50,007 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:50,672 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:51,663 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:52,690 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:53,373 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:53,673 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:54,009 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:17:54,687 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:55,675 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:56,221 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:17:56,221 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:17:56,703 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:57,697 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:58,704 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:17:59,372 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:17:59,715 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:00,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:01,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:02,012 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:02,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:03,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:04,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:05,262 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:05,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:06,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:07,698 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:08,014 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:08,704 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:09,706 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:10,262 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:10,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:10,828 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:18:11,202 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:18:11,203 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:18:11,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:12,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:13,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:14,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:15,346 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:15,708 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:16,713 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:17,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:18,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:19,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:20,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:21,262 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:21,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:22,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:23,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:24,711 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:25,712 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:26,202 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:18:26,203 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:18:26,281 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:26,712 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:27,723 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:28,022 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:28,290 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:18:28,290 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4 with policy now
2025-01-07 22:18:28,302 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:18:28,303 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:18:28,303 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:18:28,303 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:18:28,513 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/b72q6vqk-media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4
2025-01-07 22:18:28,715 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:29,023 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:18:29,023 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4
2025-01-07 22:18:29,023 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:18:29,310 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ko4djyjz-media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4
2025-01-07 22:18:29,741 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:30,023 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:30,757 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:31,024 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:31,295 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:31,750 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:32,748 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:33,767 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:34,025 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:34,770 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:35,750 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:36,026 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:36,755 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:37,291 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:37,769 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:38,757 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:39,829 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:40,027 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:18:40,142 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:18:40,142 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:18:40,143 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:18:40,774 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:40,828 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:18:41,028 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:18:41,216 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:18:41,216 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:18:41,766 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:42,028 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:42,319 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:42,774 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:43,795 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:44,775 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:45,786 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:46,030 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:46,812 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:47,791 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:48,286 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:48,783 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:49,813 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:50,837 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:51,811 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:52,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:53,416 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:53,806 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:54,033 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:18:54,806 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:55,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:56,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:18:56,208 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:18:56,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:57,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:58,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:18:59,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:18:59,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:00,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:01,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:02,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:03,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:04,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:05,277 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:05,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:06,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:07,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:08,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:09,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:10,278 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:10,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:10,829 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:19:11,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:19:11,209 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:19:11,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:12,809 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:13,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:14,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:15,361 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:15,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:16,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:17,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:18,810 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:19,839 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:20,044 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:20,474 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:20,637 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:19:20,637 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4 with policy now
2025-01-07 22:19:20,656 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:19:20,657 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:19:20,657 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:19:20,658 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:19:20,822 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:20,880 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/gweu51ng-media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4
2025-01-07 22:19:21,044 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:19:21,044 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4
2025-01-07 22:19:21,045 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:19:21,363 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/gm3ucvrw-media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4
2025-01-07 22:19:21,841 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:22,045 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:22,852 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:23,045 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:23,851 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:24,858 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:25,874 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:26,046 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:26,223 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:19:26,223 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:19:26,374 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:26,889 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:27,849 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:28,047 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:28,865 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:29,869 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:30,867 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:31,879 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:32,290 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:32,876 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:33,883 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:34,884 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:35,894 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:36,892 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:37,051 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:19:37,211 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:19:37,211 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:19:37,212 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:19:37,295 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/nvgsxp4i-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:19:37,489 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:37,896 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:38,052 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:19:38,052 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:38,908 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:39,905 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:40,829 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:19:40,891 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:41,230 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:19:41,230 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:19:41,904 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:42,053 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:42,901 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:43,356 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:43,916 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:44,902 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:45,902 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:46,898 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:47,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:48,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:49,056 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:19:49,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:49,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:50,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:51,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:52,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:53,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:54,296 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:54,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:55,899 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:56,216 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:19:56,217 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:19:56,900 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:57,900 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:58,900 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:19:59,304 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:19:59,909 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:00,909 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:01,909 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:02,910 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:03,909 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:04,910 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:05,296 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:05,910 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:06,911 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:07,911 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:08,911 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:09,911 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:10,301 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:10,830 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:20:10,911 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:11,217 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:20:11,217 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:20:11,912 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:12,912 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:13,912 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:14,936 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:15,313 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:15,417 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:20:15,417 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4 with policy now
2025-01-07 22:20:15,427 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:20:15,428 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:20:15,428 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:20:15,428 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:20:15,644 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/0qjdem3p-media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4
2025-01-07 22:20:15,928 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:16,073 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:20:16,073 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4
2025-01-07 22:20:16,073 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:20:16,360 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/3copf12h-media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4
2025-01-07 22:20:16,933 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:17,074 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:17,939 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:18,953 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:19,968 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:20,945 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:20,953 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:21,075 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:21,945 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:22,941 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:23,951 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:24,968 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:25,077 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:25,952 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:26,232 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:20:26,232 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:20:26,344 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:26,973 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:27,965 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:28,972 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:29,995 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:30,977 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:31,990 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:32,051 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:32,080 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:20:32,276 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:20:32,276 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:20:32,276 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:20:32,983 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:33,081 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:20:33,081 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:20:33,984 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:34,971 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:35,082 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:35,987 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:36,973 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:37,537 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:37,999 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:39,008 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:39,083 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:39,996 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:40,830 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:20:41,010 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:41,223 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:20:41,223 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:20:42,024 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:43,033 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:43,328 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:44,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:45,002 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:46,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:47,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:47,087 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:20:48,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:49,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:49,308 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:50,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:51,003 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:52,004 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:53,004 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:54,004 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:54,309 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:20:55,004 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:56,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:56,217 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:20:56,217 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:20:57,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:58,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:20:59,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:00,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:00,308 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:01,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:02,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:03,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:04,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:05,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:05,308 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:06,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:07,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:08,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:09,006 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:10,007 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:10,308 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:10,831 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:21:11,007 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:11,217 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:21:11,217 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:21:12,021 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:13,019 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:13,098 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:13,235 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:21:13,235 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_e673279ceb422c546539.mp4 with policy now
2025-01-07 22:21:13,241 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:21:13,248 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:21:13,248 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:21:13,248 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:21:13,461 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ohhgn9jr-media/videos/eval/17_eval_17_e673279ceb422c546539.mp4
2025-01-07 22:21:14,025 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:14,098 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:21:14,098 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/17_eval_17_e673279ceb422c546539.mp4
2025-01-07 22:21:14,099 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:21:14,386 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/stkhwb1n-media/videos/eval/17_eval_17_e673279ceb422c546539.mp4
2025-01-07 22:21:15,035 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:15,099 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:15,316 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:16,046 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:17,054 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:18,042 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:19,039 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:19,101 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:20,031 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:20,969 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:21,036 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:22,061 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:23,060 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:23,102 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:24,064 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:25,051 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:26,060 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:26,241 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:21:26,241 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:21:26,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:27,091 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:27,340 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:21:27,340 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:21:27,341 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:21:28,105 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:21:28,105 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:21:28,115 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:29,105 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:29,105 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:30,081 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:31,096 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:31,106 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:31,657 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:32,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:33,081 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:33,107 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:34,081 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:35,081 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:35,108 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:36,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:37,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:37,320 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:38,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:39,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:40,082 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:40,831 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:21:41,083 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:41,219 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:21:41,220 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:21:42,083 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:43,083 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:43,320 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:44,083 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:45,083 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:46,084 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:47,084 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:48,084 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:48,320 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:49,084 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:50,084 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:51,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:52,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:53,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:53,325 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:54,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:55,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:56,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:56,219 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:21:56,220 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:21:57,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:57,119 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:21:58,085 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:58,370 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:21:59,086 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:21:59,120 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:00,112 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:00,414 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:22:00,414 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4 with policy now
2025-01-07 22:22:00,417 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:22:00,418 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:22:00,418 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:22:00,418 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:22:00,639 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/o1vdcb9z-media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4
2025-01-07 22:22:01,102 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:01,121 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:22:01,121 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:01,121 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4
2025-01-07 22:22:01,121 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:22:01,411 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ppuctjtu-media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4
2025-01-07 22:22:02,095 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:03,095 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:03,122 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:04,096 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:04,124 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:05,116 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:06,121 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:07,124 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:07,130 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:08,125 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:09,115 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:09,124 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:09,389 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:10,112 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:10,831 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:22:11,131 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:11,226 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:22:11,226 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:22:12,124 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:12,126 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:22:12,397 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:22:12,398 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:22:12,398 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:22:13,126 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:22:13,126 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:13,143 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:14,122 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:14,721 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:15,127 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:15,150 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:16,118 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:17,121 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:17,128 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:18,121 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:19,125 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:19,129 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:20,125 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:20,352 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:21,126 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:22,126 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:23,126 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:24,126 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:25,127 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:26,127 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:26,222 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:22:26,222 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:22:26,366 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:27,127 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:28,127 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:29,133 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:30,133 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:31,133 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:32,133 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:32,341 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:33,134 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:34,133 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:35,134 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:36,134 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:37,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:38,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:38,341 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:39,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:40,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:40,832 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:22:41,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:41,138 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:41,225 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:22:41,225 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:22:42,135 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:43,139 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:43,139 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:43,764 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:44,152 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:45,140 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:45,149 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:45,374 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:22:45,374 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4 with policy now
2025-01-07 22:22:45,378 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:22:45,378 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:22:45,378 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:22:45,378 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:22:45,602 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/z1tye7wm-media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4
2025-01-07 22:22:46,141 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:22:46,141 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4
2025-01-07 22:22:46,141 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval
2025-01-07 22:22:46,192 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:46,430 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/89phxfif-media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4
2025-01-07 22:22:47,141 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:47,170 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:48,165 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:48,893 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:49,161 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:50,171 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:51,143 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:51,178 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:52,177 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:53,177 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:54,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:54,337 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:22:55,145 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:22:55,236 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:56,182 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:56,264 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:22:56,265 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:22:57,195 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:58,208 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:59,197 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:22:59,395 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:00,211 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:01,213 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:02,237 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:03,241 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:04,253 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:05,248 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:05,389 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:06,269 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:07,252 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:08,270 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:09,290 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:10,267 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:10,401 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:10,832 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:23:11,260 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:23:11,261 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:23:11,307 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:12,289 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:13,321 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:14,304 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:15,311 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:16,295 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:16,388 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:17,300 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:18,316 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:19,335 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:20,310 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:21,307 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:21,403 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:22,317 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:23,304 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:24,324 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:25,317 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:26,251 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:23:26,251 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:23:26,327 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:26,483 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:27,319 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:28,316 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:29,347 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:30,357 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:31,361 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:32,356 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:32,461 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:33,361 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:34,351 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:35,339 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:36,366 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:37,379 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:37,611 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:38,163 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:23:38,364 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:38,418 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/12cip9vv-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:23:38,502 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:23:38,502 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:38,502 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:39,164 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:39,164 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:23:39,388 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:39,520 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 22:23:39,521 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 22:23:39,883 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:23:39,931 INFO    wandb-upload_2:2240208 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpyshrfqyx
2025-01-07 22:23:40,164 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json
2025-01-07 22:23:40,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media
2025-01-07 22:23:40,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval
2025-01-07 22:23:40,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table
2025-01-07 22:23:40,328 INFO    SenderThread:2240208 [sender.py:send_request_log_artifact():1525] logged artifact run-4fmihp69-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzI0NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg5MDE1OQ==', 'latestArtifact': None}}
2025-01-07 22:23:40,328 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:23:40,328 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json with policy now
2025-01-07 22:23:40,328 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:23:40,328 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,329 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,329 DEBUG   SenderThread:2240208 [sender.py:send():379] send: telemetry
2025-01-07 22:23:40,329 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,329 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,329 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,330 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,330 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,330 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,331 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,331 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,331 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,331 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,332 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,332 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,332 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,332 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,333 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,333 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,333 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,334 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,334 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,334 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,334 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:40,335 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:40,347 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 22:23:40,347 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 22:23:40,358 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 22:23:40,359 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:40,500 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/3p0039k3-media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json
2025-01-07 22:23:40,738 INFO    wandb-upload_1:2240208 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphvl7ccyh
2025-01-07 22:23:40,761 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/ho7k69m8-media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json
2025-01-07 22:23:41,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:41,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:23:41,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json
2025-01-07 22:23:41,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval
2025-01-07 22:23:41,202 INFO    SenderThread:2240208 [sender.py:send_request_log_artifact():1525] logged artifact run-4fmihp69-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzI2NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg5MDE2OA==', 'latestArtifact': None}}
2025-01-07 22:23:41,202 DEBUG   SenderThread:2240208 [sender.py:send():379] send: files
2025-01-07 22:23:41,202 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json with policy now
2025-01-07 22:23:41,202 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:23:41,203 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:41,204 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:41,204 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:23:41,240 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 22:23:41,240 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: stop_status
2025-01-07 22:23:41,359 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:41,385 INFO    wandb-upload_0:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/p9atv43p-media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json
2025-01-07 22:23:41,643 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/3sihhq5w-media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json
2025-01-07 22:23:42,165 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:42,359 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:43,166 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:23:43,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:43,450 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:44,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:45,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:46,360 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:47,361 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:47,750 DEBUG   SenderThread:2240208 [sender.py:send():379] send: exit
2025-01-07 22:23:47,750 INFO    SenderThread:2240208 [sender.py:send_exit():586] handling exit code: 0
2025-01-07 22:23:47,751 INFO    SenderThread:2240208 [sender.py:send_exit():588] handling runtime: 1056
2025-01-07 22:23:47,751 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:47,751 INFO    SenderThread:2240208 [sender.py:send_exit():594] send defer
2025-01-07 22:23:47,751 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:47,751 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-07 22:23:47,751 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:47,752 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-07 22:23:47,752 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 1
2025-01-07 22:23:47,752 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:47,752 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-07 22:23:47,752 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:47,752 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-07 22:23:47,752 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 2
2025-01-07 22:23:47,752 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:47,752 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-07 22:23:47,752 INFO    HandlerThread:2240208 [system_monitor.py:finish():203] Stopping system monitor
2025-01-07 22:23:47,752 DEBUG   SystemMonitor:2240208 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-07 22:23:47,753 INFO    HandlerThread:2240208 [interfaces.py:finish():200] Joined cpu monitor
2025-01-07 22:23:47,753 DEBUG   SystemMonitor:2240208 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-07 22:23:47,753 INFO    HandlerThread:2240208 [interfaces.py:finish():200] Joined disk monitor
2025-01-07 22:23:47,806 INFO    HandlerThread:2240208 [interfaces.py:finish():200] Joined gpu monitor
2025-01-07 22:23:47,806 INFO    HandlerThread:2240208 [interfaces.py:finish():200] Joined memory monitor
2025-01-07 22:23:47,806 INFO    HandlerThread:2240208 [interfaces.py:finish():200] Joined network monitor
2025-01-07 22:23:47,807 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:47,807 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-07 22:23:47,807 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 3
2025-01-07 22:23:47,807 DEBUG   SenderThread:2240208 [sender.py:send():379] send: stats
2025-01-07 22:23:47,807 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:47,807 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-07 22:23:47,807 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:47,807 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-07 22:23:47,808 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 4
2025-01-07 22:23:47,808 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:47,808 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-07 22:23:48,169 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:49,169 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:23:54,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:54,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:54,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: keepalive
2025-01-07 22:23:54,509 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:54,510 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:54,510 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-07 22:23:54,510 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 5
2025-01-07 22:23:54,510 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:54,511 DEBUG   SenderThread:2240208 [sender.py:send():379] send: history
2025-01-07 22:23:54,511 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: summary_record
2025-01-07 22:23:54,511 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:54,511 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-07 22:23:54,512 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:54,512 DEBUG   SenderThread:2240208 [sender.py:send():379] send: summary
2025-01-07 22:23:54,516 INFO    SenderThread:2240208 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 22:23:54,516 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:54,516 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-07 22:23:54,516 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 6
2025-01-07 22:23:54,516 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:54,516 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-07 22:23:54,516 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:54,516 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-07 22:23:54,520 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 22:23:54,637 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 7
2025-01-07 22:23:54,638 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:54,638 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-07 22:23:54,638 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:54,638 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-07 22:23:54,747 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:55,172 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/config.yaml
2025-01-07 22:23:55,172 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:56,568 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 8
2025-01-07 22:23:56,568 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:56,568 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:56,568 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-07 22:23:56,569 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:56,569 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-07 22:23:56,569 INFO    SenderThread:2240208 [job_builder.py:build():440] Attempting to build job artifact
2025-01-07 22:23:56,572 INFO    SenderThread:2240208 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-07 22:23:56,574 INFO    SenderThread:2240208 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-07 22:23:56,589 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 9
2025-01-07 22:23:56,589 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:56,589 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-07 22:23:56,589 DEBUG   SenderThread:2240208 [sender.py:send():379] send: artifact
2025-01-07 22:23:56,747 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:56,972 INFO    wandb-upload_1:2240208 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpk6rt9n13
2025-01-07 22:23:57,068 INFO    wandb-upload_0:2240208 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpeiw4vr0r
2025-01-07 22:23:57,107 INFO    wandb-upload_2:2240208 [upload_job.py:push():88] Uploaded file /tmp/tmpphgzzj5s/wandb-job.json
2025-01-07 22:23:57,173 INFO    Thread-12 :2240208 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:23:57,646 INFO    SenderThread:2240208 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzUzMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjUyMjY3OA==', 'versionIndex': 438}}}
2025-01-07 22:23:57,646 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:57,646 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-07 22:23:57,646 INFO    SenderThread:2240208 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-07 22:23:58,174 INFO    SenderThread:2240208 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files
2025-01-07 22:23:58,174 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/config.yaml config.yaml
2025-01-07 22:23:58,175 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json wandb-summary.json
2025-01-07 22:23:58,175 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/requirements.txt requirements.txt
2025-01-07 22:23:58,175 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-metadata.json wandb-metadata.json
2025-01-07 22:23:58,175 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0 events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:23:58,175 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log output.log
2025-01-07 22:23:58,180 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/diff.patch diff.patch
2025-01-07 22:23:58,182 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4 media/videos/eval/11_eval_11_6141e16e285aea41414e.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4 media/videos/eval/15_eval_15_b314c0181db144f9c4bb.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4 media/videos/eval/0_eval_0_ae65028e83f44b9ac5b1.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/9_eval_9_c7576684fb563a572588.mp4 media/videos/eval/9_eval_9_c7576684fb563a572588.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4 media/videos/eval/2_eval_2_537ec4e4ec2357120fef.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4 media/videos/eval/19_eval_19_b857c64ab339c17adb15.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4 media/videos/eval/3_eval_3_5243c77fb64415fcde38.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4 media/videos/eval/5_eval_5_fdc3dec2e2f95a3c767d.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4 media/videos/eval/7_eval_7_0c0abd0712ae602c07c3.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4 media/videos/eval/12_eval_12_212b177b80b2034deb4f.mp4
2025-01-07 22:23:58,183 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/17_eval_17_e673279ceb422c546539.mp4 media/videos/eval/17_eval_17_e673279ceb422c546539.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4 media/videos/eval/16_eval_16_a93b394f6e07a47f91ce.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4 media/videos/eval/10_eval_10_4e9fbf635a50e60a4cb3.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4 media/videos/eval/14_eval_14_b1f7698aaf2c3d6e6abf.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4 media/videos/eval/6_eval_6_718698df80de1dc3b1e7.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4 media/videos/eval/1_eval_1_d5acd6aafe960785647d.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4 media/videos/eval/18_eval_18_fd5b9c7ce55fe4462130.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4 media/videos/eval/13_eval_13_49e329d2ff2e935251c1.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4 media/videos/eval/8_eval_8_d8cc872cbcaee100937a.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4 media/videos/eval/4_eval_4_123736b8ecd81d086c47.mp4
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json media/table/eval/total_success_rate_ci_21_54cfd2c4c0340bc54565.table.json
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json media/table/eval/success_rate_ci_20_97faedb494e69b73017e.table.json
2025-01-07 22:23:58,184 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 10
2025-01-07 22:23:58,184 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:58,184 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:58,185 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-07 22:23:58,185 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:58,185 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-07 22:23:58,185 INFO    SenderThread:2240208 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 22:23:58,353 INFO    wandb-upload_0:2240208 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/wandb-summary.json
2025-01-07 22:23:58,365 INFO    wandb-upload_1:2240208 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/config.yaml
2025-01-07 22:23:58,409 INFO    wandb-upload_2:2240208 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/requirements.txt
2025-01-07 22:23:58,450 INFO    wandb-upload_4:2240208 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/files/output.log
2025-01-07 22:23:58,526 INFO    wandb-upload_3:2240208 [upload_job.py:push():130] Uploaded file /tmp/tmp49f3m96uwandb/biyb826f-events.out.tfevents.1736305571.leonmkim-ROG-Strix-G15CS-G15CS.2240043.0
2025-01-07 22:23:58,727 INFO    Thread-11 (_thread_body):2240208 [sender.py:transition_state():614] send defer: 11
2025-01-07 22:23:58,727 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:58,727 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-07 22:23:58,727 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:58,728 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-07 22:23:58,728 INFO    SenderThread:2240208 [file_pusher.py:join():175] waiting for file pusher
2025-01-07 22:23:58,731 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 12
2025-01-07 22:23:58,731 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:58,731 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-07 22:23:58,731 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:58,731 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-07 22:23:58,731 INFO    SenderThread:2240208 [file_stream.py:finish():601] file stream finish called
2025-01-07 22:23:58,748 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:58,935 INFO    SenderThread:2240208 [file_stream.py:finish():605] file stream finish is done
2025-01-07 22:23:58,935 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 13
2025-01-07 22:23:58,936 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:58,936 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:58,936 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-07 22:23:58,936 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:58,936 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-07 22:23:58,936 INFO    SenderThread:2240208 [sender.py:transition_state():614] send defer: 14
2025-01-07 22:23:58,936 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: defer
2025-01-07 22:23:58,936 DEBUG   SenderThread:2240208 [sender.py:send():379] send: final
2025-01-07 22:23:58,937 INFO    HandlerThread:2240208 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-07 22:23:58,937 DEBUG   SenderThread:2240208 [sender.py:send():379] send: footer
2025-01-07 22:23:58,937 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: defer
2025-01-07 22:23:58,937 INFO    SenderThread:2240208 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-07 22:23:58,937 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:58,937 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: server_info
2025-01-07 22:23:58,937 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:58,938 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: server_info
2025-01-07 22:23:58,939 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 22:23:58,939 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: get_summary
2025-01-07 22:23:58,940 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-07 22:23:58,941 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 22:23:59,004 DEBUG   SenderThread:2240208 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 22:23:59,004 INFO    MainThread:2240208 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-07 22:23:59,005 INFO    MainThread:2240208 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-07 22:23:59,005 INFO    MainThread:2240208 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-07 22:23:59,005 DEBUG   HandlerThread:2240208 [handler.py:handle_request():158] handle_request: shutdown
2025-01-07 22:23:59,005 INFO    HandlerThread:2240208 [handler.py:finish():882] shutting down handler
2025-01-07 22:23:59,939 INFO    WriterThread:2240208 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/4fmihp69/wandb/run-20250107_220610-4fmihp69/run-4fmihp69.wandb
2025-01-07 22:24:00,004 INFO    SenderThread:2240208 [sender.py:finish():1615] shutting down sender
2025-01-07 22:24:00,004 INFO    SenderThread:2240208 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 22:24:00,004 INFO    SenderThread:2240208 [file_pusher.py:join():175] waiting for file pusher