2025-01-06 17:18:49,535 INFO    StreamThr :2088223 [internal.py:wandb_internal():85] W&B internal server running at pid: 2088223, started at: 2025-01-06 17:18:49.535087
2025-01-06 17:18:49,536 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status
2025-01-06 17:18:49,547 INFO    WriterThread:2088223 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/run-5w3hakcj.wandb
2025-01-06 17:18:49,548 DEBUG   SenderThread:2088223 [sender.py:send():379] send: header
2025-01-06 17:18:49,558 DEBUG   SenderThread:2088223 [sender.py:send():379] send: run
2025-01-06 17:18:49,769 INFO    SenderThread:2088223 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files
2025-01-06 17:18:49,769 INFO    SenderThread:2088223 [sender.py:_start_run_threads():1188] run started: 5w3hakcj with start time 1736201929.536184
2025-01-06 17:18:49,788 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: check_version
2025-01-06 17:18:49,788 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: check_version
2025-01-06 17:18:49,846 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: run_start
2025-01-06 17:18:49,851 DEBUG   HandlerThread:2088223 [system_info.py:__init__():26] System info init
2025-01-06 17:18:49,851 DEBUG   HandlerThread:2088223 [system_info.py:__init__():41] System info init done
2025-01-06 17:18:49,851 INFO    HandlerThread:2088223 [system_monitor.py:start():194] Starting system monitor
2025-01-06 17:18:49,851 INFO    SystemMonitor:2088223 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-06 17:18:49,851 INFO    HandlerThread:2088223 [system_monitor.py:probe():214] Collecting system info
2025-01-06 17:18:49,852 INFO    SystemMonitor:2088223 [interfaces.py:start():188] Started cpu monitoring
2025-01-06 17:18:49,852 INFO    SystemMonitor:2088223 [interfaces.py:start():188] Started disk monitoring
2025-01-06 17:18:49,853 INFO    SystemMonitor:2088223 [interfaces.py:start():188] Started gpu monitoring
2025-01-06 17:18:49,854 INFO    SystemMonitor:2088223 [interfaces.py:start():188] Started memory monitoring
2025-01-06 17:18:49,854 INFO    SystemMonitor:2088223 [interfaces.py:start():188] Started network monitoring
2025-01-06 17:18:49,866 DEBUG   HandlerThread:2088223 [system_info.py:probe():152] Probing system
2025-01-06 17:18:49,868 DEBUG   HandlerThread:2088223 [system_info.py:_probe_git():137] Probing git
2025-01-06 17:18:49,872 DEBUG   HandlerThread:2088223 [system_info.py:_probe_git():145] Probing git done
2025-01-06 17:18:49,872 DEBUG   HandlerThread:2088223 [system_info.py:probe():200] Probing system done
2025-01-06 17:18:49,872 DEBUG   HandlerThread:2088223 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-06T22:18:49.866525', 'startedAt': '2025-01-06T22:18:49.525800', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.59225, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.392, 'min': 800.0, 'max': 4700.0}, {'current': 4497.024, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 327.4358215332031}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-06 17:18:49,873 INFO    HandlerThread:2088223 [system_monitor.py:probe():224] Finished collecting system info
2025-01-06 17:18:49,873 INFO    HandlerThread:2088223 [system_monitor.py:probe():227] Publishing system info
2025-01-06 17:18:49,873 DEBUG   HandlerThread:2088223 [system_info.py:_save_code():44] Saving code
2025-01-06 17:18:49,874 DEBUG   HandlerThread:2088223 [system_info.py:_save_code():67] Saving code done
2025-01-06 17:18:49,874 DEBUG   HandlerThread:2088223 [system_info.py:_save_patches():84] Saving git patches
2025-01-06 17:18:49,936 DEBUG   HandlerThread:2088223 [system_info.py:_save_patches():126] Saving git patches done
2025-01-06 17:18:49,936 INFO    HandlerThread:2088223 [system_monitor.py:probe():229] Finished publishing system info
2025-01-06 17:18:49,939 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:18:49,939 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-06 17:18:49,939 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-06 17:18:49,940 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-06 17:18:50,165 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/gt8m6bqs-wandb-metadata.json
2025-01-06 17:18:50,185 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: python_packages
2025-01-06 17:18:50,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: python_packages
2025-01-06 17:18:50,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:18:50,188 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:18:50,199 INFO    wandb-upload_1:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/n1buykb3-code/FISH/eval_robot.py
2025-01-06 17:18:50,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:50,229 INFO    HandlerThread:2088223 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/tb"
  save: true
}
control {
  relay_id: "e26289fcd4a2434da0e8d8aba699564f"
}
_info {
  stream_id: "5w3hakcj"
}

2025-01-06 17:18:50,229 DEBUG   HandlerThread:2088223 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-06 17:18:50,234 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/e2mxru82-diff.patch
2025-01-06 17:18:50,267 DEBUG   SenderThread:2088223 [sender.py:send():379] send: telemetry
2025-01-06 17:18:50,280 DEBUG   SenderThread:2088223 [sender.py:send():379] send: tbrecord
2025-01-06 17:18:50,281 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:18:50,281 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0 with policy live
2025-01-06 17:18:50,455 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/3r60g6da-events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:18:50,771 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/diff.patch
2025-01-06 17:18:50,771 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/code/FISH/eval_robot.py
2025-01-06 17:18:50,771 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-metadata.json
2025-01-06 17:18:50,771 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/requirements.txt
2025-01-06 17:18:50,772 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:18:50,772 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:18:50,772 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/code/FISH
2025-01-06 17:18:50,772 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/code
2025-01-06 17:18:51,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:52,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:52,770 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:18:53,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:54,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:55,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:18:55,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:56,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:57,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:58,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:59,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:18:59,899 DEBUG   SenderThread:2088223 [sender.py:send():379] send: config
2025-01-06 17:19:00,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:00,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:01,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:02,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:02,773 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:03,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:04,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:04,774 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:05,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:19:05,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:19:05,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:05,295 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:06,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:06,774 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:07,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:08,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:08,775 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:09,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:10,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:11,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:11,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:12,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:13,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:14,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:15,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:16,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:16,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:17,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:18,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:19,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:20,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:19:20,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:19:20,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:21,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:21,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:21,778 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/config.yaml
2025-01-06 17:19:22,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:23,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:24,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:25,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:26,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:26,521 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:26,780 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:27,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:28,233 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:29,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:30,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:30,781 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:31,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:32,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:32,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:33,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:34,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:34,782 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:35,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:19:35,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:19:35,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:36,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:37,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:37,291 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:38,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:39,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:40,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:41,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:42,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:43,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:43,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:44,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:45,203 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:46,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:47,206 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:48,188 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:48,206 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:49,206 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:49,855 DEBUG   SystemMonitor:2088223 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-06 17:19:49,855 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:19:50,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:19:50,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:19:50,206 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:50,795 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:51,206 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:52,207 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:53,207 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:53,236 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:54,207 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:55,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:56,122 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:19:56,123 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4 with policy now
2025-01-06 17:19:56,123 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:19:56,124 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:19:56,124 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:19:56,124 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:19:56,207 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:56,329 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/9g4zrg69-media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos
2025-01-06 17:19:56,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval
2025-01-06 17:19:57,059 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/0jamwfhf-media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4
2025-01-06 17:19:57,207 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:58,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:19:58,797 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:19:59,131 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:19:59,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:00,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:00,798 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:01,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:02,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:03,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:04,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:04,442 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:04,799 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:04,800 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:20:05,049 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/5e0faf8q-events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:20:05,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:20:05,186 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:20:05,208 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:06,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:06,800 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:07,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:08,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:09,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:10,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:10,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:10,802 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:11,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:12,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:13,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:14,209 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:15,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:15,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:16,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:17,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:18,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:19,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:19,856 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:20:20,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:20:20,187 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:20:20,233 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:20,245 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:21,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:22,210 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:23,211 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:24,211 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:25,211 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:26,189 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:26,211 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:27,211 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:28,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:29,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:30,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:31,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:31,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:32,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:33,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:34,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:35,186 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:20:35,187 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:20:35,212 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:36,204 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:20:36,205 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4 with policy now
2025-01-06 17:20:36,205 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:36,205 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:20:36,206 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:20:36,206 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:20:36,208 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:20:36,215 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:36,412 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/4r8at4h6-media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4
2025-01-06 17:20:36,810 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:20:36,810 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:36,810 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4
2025-01-06 17:20:36,810 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval
2025-01-06 17:20:37,068 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/0w05fpom-media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4
2025-01-06 17:20:37,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:38,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:38,810 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:39,227 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:40,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:40,811 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:41,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:42,194 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:42,217 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:43,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:44,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:44,812 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:45,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:46,217 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:46,813 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:20:47,216 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:47,506 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:47,641 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:20:47,641 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:20:47,641 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:20:47,813 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:20:48,217 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:48,814 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:49,217 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:49,857 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:20:50,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:20:50,188 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:20:50,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:51,218 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:52,218 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:52,815 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:53,218 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:53,484 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:54,218 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:55,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:56,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:57,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:58,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:20:58,817 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:20:58,861 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:20:59,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:00,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:01,219 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:02,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:03,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:04,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:04,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:05,187 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:21:05,187 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:21:05,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:06,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:07,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:08,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:09,220 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:10,077 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:10,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:10,821 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:11,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:12,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:13,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:14,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:15,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:15,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:16,221 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:17,233 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:18,223 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:18,413 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:21:18,413 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:21:18,414 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4 with policy now
2025-01-06 17:21:18,414 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:21:18,414 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:21:18,415 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:21:18,631 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/6xakmz00-media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4
2025-01-06 17:21:18,824 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:21:18,824 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:18,824 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4
2025-01-06 17:21:18,824 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval
2025-01-06 17:21:19,078 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/fcewp0dk-media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4
2025-01-06 17:21:19,226 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:19,857 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:21:20,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:21:20,193 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:21:20,237 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:20,270 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:20,825 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:21,234 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:22,229 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:23,235 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:24,233 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:24,826 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:25,232 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:26,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:26,234 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:26,827 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:27,237 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:28,239 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:29,242 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:30,242 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:31,199 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:31,242 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:32,246 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:33,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:34,248 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:34,829 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:21:35,070 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/fjcxvmgt-events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:21:35,191 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:21:35,191 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:21:35,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:35,479 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:21:35,479 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:21:35,479 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:21:35,830 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:21:36,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:36,785 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:36,830 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:37,255 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:38,254 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:39,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:40,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:40,831 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:21:41,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:42,194 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:42,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:43,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:44,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:45,249 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:46,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:47,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:48,194 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:48,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:49,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:49,858 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:21:50,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:21:50,190 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:21:50,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:51,250 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:52,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:53,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:53,325 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:54,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:55,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:56,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:57,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:58,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:21:59,194 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:21:59,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:00,251 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:01,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:02,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:03,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:04,195 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:04,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:05,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:22:05,190 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:22:05,252 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:06,255 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:06,840 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:07,141 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:22:07,141 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4 with policy now
2025-01-06 17:22:07,160 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:22:07,160 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:22:07,160 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:22:07,161 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:22:07,257 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:07,346 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/1lcx2tqf-media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4
2025-01-06 17:22:07,841 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:22:07,841 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4
2025-01-06 17:22:07,841 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval
2025-01-06 17:22:08,101 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/0pavcjfu-media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4
2025-01-06 17:22:08,261 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:08,841 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:09,262 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:09,841 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:10,192 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:10,261 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:11,259 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:12,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:12,842 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:13,264 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:14,262 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:14,843 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:15,197 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:15,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:16,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:17,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:18,263 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:19,261 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:19,530 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:22:19,530 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:22:19,530 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:22:19,845 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:22:19,845 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:22:19,858 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:22:20,193 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:22:20,193 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:22:20,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:20,341 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:20,845 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:21,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:22,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:22,846 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:23,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:24,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:24,847 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:25,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:26,196 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:26,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:26,847 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:27,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:28,265 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:29,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:30,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:31,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:32,196 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:32,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:33,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:34,266 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:35,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:22:35,190 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:22:35,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:36,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:37,240 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:37,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:38,267 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:39,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:40,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:41,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:42,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:43,196 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:43,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:44,268 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:45,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:46,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:47,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:48,197 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:48,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:49,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:49,859 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:22:50,190 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:22:50,190 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:22:50,269 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:51,273 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:51,912 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:22:51,913 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4 with policy now
2025-01-06 17:22:51,917 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:22:51,918 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:22:51,918 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:22:51,918 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:22:52,130 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/9k26rq02-media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4
2025-01-06 17:22:52,272 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:52,856 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:22:52,857 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:52,857 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4
2025-01-06 17:22:52,857 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval
2025-01-06 17:22:53,129 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/x4lwqnba-media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4
2025-01-06 17:22:53,281 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:53,936 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:22:54,280 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:55,279 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:56,280 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:56,858 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:22:57,286 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:58,283 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:59,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:22:59,377 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:23:00,289 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:00,859 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:01,288 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:02,294 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:03,299 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:04,293 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:05,202 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 17:23:05,202 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: stop_status
2025-01-06 17:23:05,276 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:23:05,291 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:05,861 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:23:06,112 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/epgz5fhh-events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:23:06,280 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 17:23:06,280 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 17:23:06,294 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:06,460 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:23:06,725 INFO    wandb-upload_0:2088223 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpt9ubxmqi
2025-01-06 17:23:06,809 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 17:23:06,822 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table
2025-01-06 17:23:06,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table/eval
2025-01-06 17:23:07,102 INFO    SenderThread:2088223 [sender.py:send_request_log_artifact():1525] logged artifact run-5w3hakcj-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTM0Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxMDU5Mw==', 'latestArtifact': None}}
2025-01-06 17:23:07,102 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:23:07,102 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now
2025-01-06 17:23:07,104 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:23:07,104 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,104 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,104 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:23:07,105 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,108 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,108 DEBUG   SenderThread:2088223 [sender.py:send():379] send: telemetry
2025-01-06 17:23:07,109 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,109 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,109 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,109 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,109 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,110 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,110 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,110 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,110 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,111 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,111 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,111 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,111 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,112 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,112 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,112 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,112 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,113 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,113 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,113 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,113 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,113 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,114 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 17:23:07,267 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/b8bdhkrj-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json
2025-01-06 17:23:07,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:07,324 INFO    wandb-upload_1:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/kirrr60p-media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json
2025-01-06 17:23:07,484 INFO    wandb-upload_0:2088223 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp_goqsqj8
2025-01-06 17:23:07,514 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/xs3f6cik-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json
2025-01-06 17:23:07,845 INFO    SenderThread:2088223 [sender.py:send_request_log_artifact():1525] logged artifact run-5w3hakcj-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTM1OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxMDU5NQ==', 'latestArtifact': None}}
2025-01-06 17:23:07,845 DEBUG   SenderThread:2088223 [sender.py:send():379] send: files
2025-01-06 17:23:07,845 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json with policy now
2025-01-06 17:23:07,846 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:23:07,846 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:07,846 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:07,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:23:08,084 INFO    wandb-upload_1:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/khasfmu7-media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json
2025-01-06 17:23:08,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:08,862 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:09,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:10,290 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:10,459 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:23:10,566 DEBUG   SenderThread:2088223 [sender.py:send():379] send: exit
2025-01-06 17:23:10,567 INFO    SenderThread:2088223 [sender.py:send_exit():586] handling exit code: 0
2025-01-06 17:23:10,567 INFO    SenderThread:2088223 [sender.py:send_exit():588] handling runtime: 260
2025-01-06 17:23:10,568 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:10,568 INFO    SenderThread:2088223 [sender.py:send_exit():594] send defer
2025-01-06 17:23:10,568 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:10,568 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-06 17:23:10,568 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:10,568 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-06 17:23:10,568 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 1
2025-01-06 17:23:10,568 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:10,569 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-06 17:23:10,569 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:10,569 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-06 17:23:10,569 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 2
2025-01-06 17:23:10,569 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:10,569 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-06 17:23:10,569 INFO    HandlerThread:2088223 [system_monitor.py:finish():203] Stopping system monitor
2025-01-06 17:23:10,569 DEBUG   SystemMonitor:2088223 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-06 17:23:10,569 DEBUG   SystemMonitor:2088223 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-06 17:23:10,570 INFO    HandlerThread:2088223 [interfaces.py:finish():200] Joined cpu monitor
2025-01-06 17:23:10,570 INFO    HandlerThread:2088223 [interfaces.py:finish():200] Joined disk monitor
2025-01-06 17:23:10,623 INFO    HandlerThread:2088223 [interfaces.py:finish():200] Joined gpu monitor
2025-01-06 17:23:10,623 INFO    HandlerThread:2088223 [interfaces.py:finish():200] Joined memory monitor
2025-01-06 17:23:10,624 INFO    HandlerThread:2088223 [interfaces.py:finish():200] Joined network monitor
2025-01-06 17:23:10,624 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:10,624 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-06 17:23:10,624 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 3
2025-01-06 17:23:10,624 DEBUG   SenderThread:2088223 [sender.py:send():379] send: stats
2025-01-06 17:23:10,624 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:10,624 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-06 17:23:10,625 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:10,625 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-06 17:23:10,625 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 4
2025-01-06 17:23:10,625 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:10,625 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-06 17:23:10,863 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/config.yaml
2025-01-06 17:23:10,863 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:23:12,864 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:17,592 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 17:23:17,592 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:17,592 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:23:17,593 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: keepalive
2025-01-06 17:23:17,593 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-06 17:23:17,594 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 5
2025-01-06 17:23:17,594 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 17:23:17,594 DEBUG   SenderThread:2088223 [sender.py:send():379] send: history
2025-01-06 17:23:17,594 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:17,594 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-06 17:23:17,594 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: summary_record
2025-01-06 17:23:17,595 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:17,595 DEBUG   SenderThread:2088223 [sender.py:send():379] send: summary
2025-01-06 17:23:17,596 INFO    SenderThread:2088223 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 17:23:17,596 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:17,596 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-06 17:23:17,596 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 6
2025-01-06 17:23:17,596 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:17,596 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-06 17:23:17,596 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:17,596 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-06 17:23:17,596 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 7
2025-01-06 17:23:17,596 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 17:23:17,597 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:17,597 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-06 17:23:17,597 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:17,597 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-06 17:23:17,866 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:23:18,503 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 17:23:19,077 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 8
2025-01-06 17:23:19,077 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 17:23:19,077 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:19,078 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-06 17:23:19,078 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:19,078 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-06 17:23:19,078 INFO    SenderThread:2088223 [job_builder.py:build():440] Attempting to build job artifact
2025-01-06 17:23:19,080 INFO    SenderThread:2088223 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-06 17:23:19,082 INFO    SenderThread:2088223 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-06 17:23:19,090 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 9
2025-01-06 17:23:19,091 DEBUG   SenderThread:2088223 [sender.py:send():379] send: artifact
2025-01-06 17:23:19,093 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:19,093 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-06 17:23:19,450 INFO    wandb-upload_1:2088223 [upload_job.py:push():85] Skipped uploading /tmp/tmpxdqw67u_/wandb-job.json
2025-01-06 17:23:19,450 INFO    wandb-upload_0:2088223 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdz6bjyvd
2025-01-06 17:23:19,503 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 17:23:19,564 INFO    wandb-upload_2:2088223 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp49htejn7
2025-01-06 17:23:19,867 INFO    Thread-12 :2088223 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:20,102 INFO    SenderThread:2088223 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTY1Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjU3MQ==', 'versionIndex': 422}}}
2025-01-06 17:23:20,102 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:20,102 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-06 17:23:20,102 INFO    SenderThread:2088223 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-06 17:23:20,868 INFO    SenderThread:2088223 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files
2025-01-06 17:23:20,868 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/config.yaml config.yaml
2025-01-06 17:23:20,869 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json wandb-summary.json
2025-01-06 17:23:20,869 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0 events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:23:20,871 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/requirements.txt requirements.txt
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-metadata.json wandb-metadata.json
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log output.log
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/diff.patch diff.patch
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4 media/videos/eval/1_eval_1_caf44f191ae0ae9f7dee.mp4
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4 media/videos/eval/0_eval_0_618aacdea903d28c41fe.mp4
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4 media/videos/eval/4_eval_4_4607322fe9bf46c96947.mp4
2025-01-06 17:23:20,873 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4 media/videos/eval/2_eval_2_493dcb44b239a781188a.mp4
2025-01-06 17:23:20,874 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4 media/videos/eval/3_eval_3_b633df55c1c901ea4269.mp4
2025-01-06 17:23:20,874 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json media/table/eval/total_success_rate_ci_6_e3405f8336f17a4a59dd.table.json
2025-01-06 17:23:20,875 INFO    SenderThread:2088223 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json
2025-01-06 17:23:20,875 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 10
2025-01-06 17:23:20,875 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 17:23:20,876 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:20,880 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-06 17:23:20,880 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:20,880 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-06 17:23:20,880 INFO    SenderThread:2088223 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 17:23:21,039 INFO    wandb-upload_2:2088223 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/wandb-summary.json
2025-01-06 17:23:21,056 INFO    wandb-upload_1:2088223 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/config.yaml
2025-01-06 17:23:21,102 INFO    wandb-upload_4:2088223 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/output.log
2025-01-06 17:23:21,120 INFO    wandb-upload_0:2088223 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/files/requirements.txt
2025-01-06 17:23:21,143 INFO    wandb-upload_3:2088223 [upload_job.py:push():130] Uploaded file /tmp/tmpg7sw7vxxwandb/aw4645rl-events.out.tfevents.1736201930.leonmkim-ROG-Strix-G15CS-G15CS.2088037.0
2025-01-06 17:23:21,343 INFO    Thread-11 (_thread_body):2088223 [sender.py:transition_state():614] send defer: 11
2025-01-06 17:23:21,343 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:21,343 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-06 17:23:21,343 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:21,343 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-06 17:23:21,344 INFO    SenderThread:2088223 [file_pusher.py:join():175] waiting for file pusher
2025-01-06 17:23:21,345 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 12
2025-01-06 17:23:21,345 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:21,345 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-06 17:23:21,345 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:21,345 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-06 17:23:21,345 INFO    SenderThread:2088223 [file_stream.py:finish():601] file stream finish called
2025-01-06 17:23:21,406 INFO    SenderThread:2088223 [file_stream.py:finish():605] file stream finish is done
2025-01-06 17:23:21,407 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 13
2025-01-06 17:23:21,407 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:21,407 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-06 17:23:21,407 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:21,407 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-06 17:23:21,407 INFO    SenderThread:2088223 [sender.py:transition_state():614] send defer: 14
2025-01-06 17:23:21,407 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: defer
2025-01-06 17:23:21,407 INFO    HandlerThread:2088223 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-06 17:23:21,407 DEBUG   SenderThread:2088223 [sender.py:send():379] send: final
2025-01-06 17:23:21,408 DEBUG   SenderThread:2088223 [sender.py:send():379] send: footer
2025-01-06 17:23:21,408 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: defer
2025-01-06 17:23:21,408 INFO    SenderThread:2088223 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-06 17:23:21,408 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 17:23:21,408 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 17:23:21,408 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: server_info
2025-01-06 17:23:21,409 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 17:23:21,409 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: server_info
2025-01-06 17:23:21,410 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: get_summary
2025-01-06 17:23:21,411 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-06 17:23:21,411 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 17:23:21,465 DEBUG   SenderThread:2088223 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 17:23:21,465 INFO    MainThread:2088223 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-06 17:23:21,465 INFO    MainThread:2088223 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-06 17:23:21,466 INFO    MainThread:2088223 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-06 17:23:21,466 DEBUG   HandlerThread:2088223 [handler.py:handle_request():158] handle_request: shutdown
2025-01-06 17:23:21,466 INFO    HandlerThread:2088223 [handler.py:finish():882] shutting down handler
2025-01-06 17:23:22,409 INFO    WriterThread:2088223 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/5w3hakcj/wandb/run-20250106_171849-5w3hakcj/run-5w3hakcj.wandb
2025-01-06 17:23:22,465 INFO    SenderThread:2088223 [sender.py:finish():1615] shutting down sender
2025-01-06 17:23:22,465 INFO    SenderThread:2088223 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 17:23:22,465 INFO    SenderThread:2088223 [file_pusher.py:join():175] waiting for file pusher