2025-01-06 22:46:34,435 INFO    StreamThr :2129923 [internal.py:wandb_internal():85] W&B internal server running at pid: 2129923, started at: 2025-01-06 22:46:34.434866
2025-01-06 22:46:34,436 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status
2025-01-06 22:46:34,447 INFO    WriterThread:2129923 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/run-bbomq0jb.wandb
2025-01-06 22:46:34,448 DEBUG   SenderThread:2129923 [sender.py:send():379] send: header
2025-01-06 22:46:34,458 DEBUG   SenderThread:2129923 [sender.py:send():379] send: run
2025-01-06 22:46:34,695 INFO    SenderThread:2129923 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files
2025-01-06 22:46:34,695 INFO    SenderThread:2129923 [sender.py:_start_run_threads():1188] run started: bbomq0jb with start time 1736221594.435095
2025-01-06 22:46:34,716 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: check_version
2025-01-06 22:46:34,716 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: check_version
2025-01-06 22:46:34,763 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: run_start
2025-01-06 22:46:34,768 DEBUG   HandlerThread:2129923 [system_info.py:__init__():26] System info init
2025-01-06 22:46:34,768 DEBUG   HandlerThread:2129923 [system_info.py:__init__():41] System info init done
2025-01-06 22:46:34,768 INFO    HandlerThread:2129923 [system_monitor.py:start():194] Starting system monitor
2025-01-06 22:46:34,768 INFO    SystemMonitor:2129923 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-06 22:46:34,769 INFO    HandlerThread:2129923 [system_monitor.py:probe():214] Collecting system info
2025-01-06 22:46:34,769 INFO    SystemMonitor:2129923 [interfaces.py:start():188] Started cpu monitoring
2025-01-06 22:46:34,770 INFO    SystemMonitor:2129923 [interfaces.py:start():188] Started disk monitoring
2025-01-06 22:46:34,770 INFO    SystemMonitor:2129923 [interfaces.py:start():188] Started gpu monitoring
2025-01-06 22:46:34,771 INFO    SystemMonitor:2129923 [interfaces.py:start():188] Started memory monitoring
2025-01-06 22:46:34,771 INFO    SystemMonitor:2129923 [interfaces.py:start():188] Started network monitoring
2025-01-06 22:46:34,781 DEBUG   HandlerThread:2129923 [system_info.py:probe():152] Probing system
2025-01-06 22:46:34,783 DEBUG   HandlerThread:2129923 [system_info.py:_probe_git():137] Probing git
2025-01-06 22:46:34,788 DEBUG   HandlerThread:2129923 [system_info.py:_probe_git():145] Probing git done
2025-01-06 22:46:34,788 DEBUG   HandlerThread:2129923 [system_info.py:probe():200] Probing system done
2025-01-06 22:46:34,788 DEBUG   HandlerThread:2129923 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T03:46:34.781844', 'startedAt': '2025-01-07T03:46:34.427712', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3937.05225, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4498.855, 'min': 800.0, 'max': 4700.0}, {'current': 4498.576, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.287, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}, {'current': 4499.963, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 614.3022308349609}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-06 22:46:34,788 INFO    HandlerThread:2129923 [system_monitor.py:probe():224] Finished collecting system info
2025-01-06 22:46:34,788 INFO    HandlerThread:2129923 [system_monitor.py:probe():227] Publishing system info
2025-01-06 22:46:34,788 DEBUG   HandlerThread:2129923 [system_info.py:_save_code():44] Saving code
2025-01-06 22:46:34,789 DEBUG   HandlerThread:2129923 [system_info.py:_save_code():67] Saving code done
2025-01-06 22:46:34,789 DEBUG   HandlerThread:2129923 [system_info.py:_save_patches():84] Saving git patches
2025-01-06 22:46:34,854 DEBUG   HandlerThread:2129923 [system_info.py:_save_patches():126] Saving git patches done
2025-01-06 22:46:34,854 INFO    HandlerThread:2129923 [system_monitor.py:probe():229] Finished publishing system info
2025-01-06 22:46:34,857 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:46:34,857 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-06 22:46:34,858 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-06 22:46:34,858 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-06 22:46:35,093 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/glhoyajj-wandb-metadata.json
2025-01-06 22:46:35,104 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: python_packages
2025-01-06 22:46:35,104 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: python_packages
2025-01-06 22:46:35,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:46:35,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:46:35,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:35,106 INFO    HandlerThread:2129923 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/tb"
  save: true
}
control {
  relay_id: "cfd43c4364754628a34595dcf6569b62"
}
_info {
  stream_id: "bbomq0jb"
}

2025-01-06 22:46:35,107 DEBUG   HandlerThread:2129923 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-06 22:46:35,127 INFO    wandb-upload_2:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/mco025zc-diff.patch
2025-01-06 22:46:35,132 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/6qzbs4ta-code/FISH/eval_robot.py
2025-01-06 22:46:35,164 DEBUG   SenderThread:2129923 [sender.py:send():379] send: telemetry
2025-01-06 22:46:35,164 DEBUG   SenderThread:2129923 [sender.py:send():379] send: tbrecord
2025-01-06 22:46:35,164 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:46:35,164 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0 with policy live
2025-01-06 22:46:35,335 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/me5c1w3s-events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:46:35,697 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-metadata.json
2025-01-06 22:46:35,697 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/diff.patch
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/code/FISH/eval_robot.py
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/requirements.txt
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/code/FISH
2025-01-06 22:46:35,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/code
2025-01-06 22:46:36,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:37,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:37,696 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:38,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:39,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:39,660 DEBUG   SenderThread:2129923 [sender.py:send():379] send: config
2025-01-06 22:46:39,660 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:46:40,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:41,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:41,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:42,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:43,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:43,698 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:44,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:45,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:46:45,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:45,699 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:46,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:47,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:48,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:49,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:49,700 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:50,104 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:46:50,104 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:46:50,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:50,196 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:46:51,109 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:52,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:53,109 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:53,701 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:54,109 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:55,109 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:55,702 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:55,827 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:46:56,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:57,114 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:57,703 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:46:58,114 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:46:59,117 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:00,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:01,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:01,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:02,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:03,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:04,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:05,104 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:47:05,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:47:05,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:06,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:06,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:06,705 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/config.yaml
2025-01-06 22:47:07,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:08,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:09,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:10,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:11,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:11,960 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:12,116 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:13,117 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:13,707 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:14,118 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:15,117 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:15,707 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:16,118 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:17,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:17,118 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:18,118 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:19,118 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:20,104 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:47:20,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:47:20,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:21,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:22,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:22,304 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:23,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:24,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:25,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:26,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:27,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:28,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:28,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:29,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:30,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:31,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:31,516 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:47:31,516 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4 with policy now
2025-01-06 22:47:31,516 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:47:31,517 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:47:31,518 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:47:31,518 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos
2025-01-06 22:47:31,712 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media
2025-01-06 22:47:31,777 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/0wxi5wtb-media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4
2025-01-06 22:47:32,044 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/eeh36p89-media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4
2025-01-06 22:47:32,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:33,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:33,538 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:33,713 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:34,120 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:34,771 DEBUG   SystemMonitor:2129923 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-06 22:47:34,772 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:47:35,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:47:35,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:47:35,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:36,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:37,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:37,714 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:38,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:38,901 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:39,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:39,715 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:40,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:41,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:42,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:43,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:44,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:44,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:45,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:46,121 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:47,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:48,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:49,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:49,718 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:47:49,853 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:49,996 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/huzfsfha-events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:47:50,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:47:50,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:47:50,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:51,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:51,718 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:52,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:53,122 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:53,719 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:54,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:55,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:47:55,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:55,720 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:47:56,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:57,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:58,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:47:59,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:00,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:00,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:01,123 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:02,124 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:03,124 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:04,124 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:04,772 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:48:05,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:48:05,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:48:05,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:05,184 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:06,124 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:07,126 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:07,724 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:08,126 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:09,126 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:10,126 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:11,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:11,126 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:12,127 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:13,127 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:14,127 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:15,128 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:16,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:16,128 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:17,128 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:18,128 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:19,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:20,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:48:20,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:48:20,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:21,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:21,221 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:22,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:23,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:24,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:25,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:26,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:27,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:27,129 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:28,130 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:28,676 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:48:28,676 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4 with policy now
2025-01-06 22:48:28,679 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:48:28,679 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:48:28,679 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:48:28,684 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:48:28,731 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:48:28,732 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4
2025-01-06 22:48:28,732 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval
2025-01-06 22:48:28,936 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/mgba4a4g-media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4
2025-01-06 22:48:29,130 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:29,208 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/o37dwl2w-media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4
2025-01-06 22:48:29,732 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:30,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:30,732 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:31,138 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:32,110 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:32,140 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:33,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:33,733 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:34,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:34,773 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:48:35,108 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:48:35,108 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:48:35,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:36,139 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:37,139 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:37,245 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:37,734 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:38,139 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:39,139 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:40,140 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:41,142 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:42,142 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:43,110 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:43,142 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:44,142 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:45,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:46,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:47,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:47,540 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:48:47,540 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:48:47,541 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:48:47,738 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:48:47,738 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:48:48,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:49,037 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:49,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:49,738 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:50,105 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:48:50,105 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:48:50,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:51,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:51,739 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:52,143 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:53,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:53,740 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:48:54,114 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:48:54,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:55,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:56,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:57,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:58,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:48:59,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:00,110 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:00,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:01,144 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:02,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:03,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:04,145 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:04,773 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:49:05,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:49:05,106 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:49:05,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:05,216 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:06,147 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:07,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:08,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:09,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:10,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:11,111 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:11,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:12,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:13,146 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:14,147 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:15,147 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:16,111 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:16,147 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:17,147 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:18,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:19,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:20,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:49:20,106 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:49:20,149 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:21,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:21,195 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:22,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:23,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:24,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:25,148 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:25,751 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:26,150 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:26,445 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:26,922 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:49:26,922 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4 with policy now
2025-01-06 22:49:26,924 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:49:26,925 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:49:26,925 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:49:26,925 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:49:27,151 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:27,172 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/u2gyat7g-media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4
2025-01-06 22:49:27,751 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:49:27,751 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:27,752 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4
2025-01-06 22:49:27,752 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval
2025-01-06 22:49:28,045 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/xc5pw4o9-media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4
2025-01-06 22:49:28,154 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:29,156 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:29,752 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:30,156 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:31,154 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:31,593 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:31,753 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:32,153 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:33,155 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:34,154 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:34,774 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:49:35,110 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:49:35,110 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:49:35,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:35,754 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:36,158 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:36,371 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:49:36,371 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:49:36,371 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:49:36,754 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:49:36,755 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:49:36,998 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/12hzf77q-events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:49:37,158 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:37,409 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:37,755 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:38,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:39,158 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:39,755 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:40,197 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:41,163 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:41,757 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:42,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:42,696 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:43,159 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:43,758 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:44,159 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:45,159 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:46,159 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:47,159 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:47,759 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:49:47,833 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:48,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:49,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:50,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:49:50,106 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:49:50,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:51,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:52,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:53,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:53,226 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:54,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:55,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:56,160 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:57,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:58,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:59,009 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:49:59,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:49:59,763 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:00,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:01,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:02,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:03,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:04,113 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:04,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:04,774 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:50:05,106 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:50:05,107 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:50:05,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:06,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:07,161 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:08,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:09,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:09,231 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:10,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:11,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:12,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:13,162 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:14,163 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:15,113 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:15,163 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:16,193 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:17,167 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:17,317 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:50:17,317 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4 with policy now
2025-01-06 22:50:17,318 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:50:17,319 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:50:17,319 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:50:17,341 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:50:17,582 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/unzssuoz-media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4
2025-01-06 22:50:17,769 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:50:17,770 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:17,770 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4
2025-01-06 22:50:17,770 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval
2025-01-06 22:50:18,038 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/sn86bu0l-media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4
2025-01-06 22:50:18,172 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:19,182 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:19,770 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:20,109 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:50:20,109 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:50:20,175 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:20,184 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:21,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:21,771 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:22,179 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:23,183 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:24,181 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:25,193 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:25,772 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:26,119 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:26,181 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:27,183 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:27,438 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:50:27,438 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:50:27,438 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:50:27,773 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:27,773 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:50:27,773 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:50:28,183 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:29,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:29,774 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:30,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:31,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:31,774 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:31,983 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:32,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:33,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:33,775 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:34,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:34,775 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:50:35,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:50:35,107 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:50:35,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:36,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:37,184 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:37,185 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:38,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:39,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:40,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:41,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:42,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:43,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:43,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:44,186 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:45,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:45,779 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:46,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:47,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:47,780 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:48,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:48,313 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:49,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:50,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:50:50,107 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:50:50,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:51,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:52,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:53,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:54,187 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:54,249 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:50:55,188 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:56,213 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:57,188 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:57,783 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:50:58,189 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:50:59,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:00,115 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:00,189 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:01,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:02,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:03,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:04,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:04,775 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:51:05,107 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:51:05,107 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:51:05,190 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:05,208 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:06,192 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:06,943 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:51:06,943 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_2e71831cf69946749641.mp4 with policy now
2025-01-06 22:51:06,945 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:51:06,946 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:51:06,946 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:07,043 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:07,193 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:07,199 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/ptl12dj7-media/videos/eval/4_eval_4_2e71831cf69946749641.mp4
2025-01-06 22:51:07,787 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:07,788 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:07,788 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/4_eval_4_2e71831cf69946749641.mp4
2025-01-06 22:51:07,788 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval
2025-01-06 22:51:08,071 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/3dnhozdf-media/videos/eval/4_eval_4_2e71831cf69946749641.mp4
2025-01-06 22:51:08,197 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:09,206 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:09,788 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:10,207 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:10,382 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:11,214 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:11,789 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:12,216 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:13,213 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:14,206 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:15,206 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:15,790 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:16,131 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:16,211 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:17,206 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:18,211 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:19,207 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:20,114 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 22:51:20,114 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: stop_status
2025-01-06 22:51:20,210 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:20,502 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:51:20,502 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:20,505 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:20,562 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 22:51:20,562 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 22:51:20,711 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:51:20,792 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:20,792 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:51:20,792 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json
2025-01-06 22:51:20,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media
2025-01-06 22:51:20,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table
2025-01-06 22:51:20,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval
2025-01-06 22:51:21,033 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/lphlpuz1-media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json
2025-01-06 22:51:21,034 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/vnlh6jns-events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:51:21,057 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 22:51:21,065 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 22:51:21,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:21,286 INFO    wandb-upload_2:2129923 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp7petc0ni
2025-01-06 22:51:21,774 INFO    SenderThread:2129923 [sender.py:send_request_log_artifact():1525] logged artifact run-bbomq0jb-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjY0NDUwMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc2NDEwMQ==', 'latestArtifact': None}}
2025-01-06 22:51:21,774 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:51:21,774 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json with policy now
2025-01-06 22:51:21,775 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:21,775 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:51:21,775 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,775 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,776 DEBUG   SenderThread:2129923 [sender.py:send():379] send: telemetry
2025-01-06 22:51:21,776 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,779 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,780 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,780 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,780 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,780 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,781 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,781 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,781 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,781 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,782 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,782 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,782 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,782 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,783 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,783 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,783 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,783 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,784 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,784 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,784 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:21,784 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:21,785 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 22:51:21,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:21,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:21,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:51:21,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json
2025-01-06 22:51:21,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval
2025-01-06 22:51:21,996 INFO    wandb-upload_2:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/1k2tyvef-media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json
2025-01-06 22:51:22,024 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/c4or21u1-media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json
2025-01-06 22:51:22,141 INFO    wandb-upload_0:2129923 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphp0rib_k
2025-01-06 22:51:22,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:22,496 INFO    SenderThread:2129923 [sender.py:send_request_log_artifact():1525] logged artifact run-bbomq0jb-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjY0NDUzMg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc2NDEwMw==', 'latestArtifact': None}}
2025-01-06 22:51:22,496 DEBUG   SenderThread:2129923 [sender.py:send():379] send: files
2025-01-06 22:51:22,496 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json with policy now
2025-01-06 22:51:22,497 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:51:22,497 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:22,497 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:22,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:22,794 INFO    wandb-upload_2:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/y7f5u63o-media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json
2025-01-06 22:51:23,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:23,793 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:24,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:25,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:26,204 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:27,125 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:27,205 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:27,795 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/config.yaml
2025-01-06 22:51:28,205 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:29,205 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:29,693 DEBUG   SenderThread:2129923 [sender.py:send():379] send: exit
2025-01-06 22:51:29,693 INFO    SenderThread:2129923 [sender.py:send_exit():586] handling exit code: 0
2025-01-06 22:51:29,693 INFO    SenderThread:2129923 [sender.py:send_exit():588] handling runtime: 294
2025-01-06 22:51:29,694 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:29,694 INFO    SenderThread:2129923 [sender.py:send_exit():594] send defer
2025-01-06 22:51:29,694 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:29,694 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-06 22:51:29,694 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:29,694 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-06 22:51:29,694 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 1
2025-01-06 22:51:29,694 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:29,694 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-06 22:51:29,695 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:29,695 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-06 22:51:29,695 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 2
2025-01-06 22:51:29,695 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:29,695 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-06 22:51:29,695 INFO    HandlerThread:2129923 [system_monitor.py:finish():203] Stopping system monitor
2025-01-06 22:51:29,695 DEBUG   SystemMonitor:2129923 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-06 22:51:29,695 DEBUG   SystemMonitor:2129923 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-06 22:51:29,696 INFO    HandlerThread:2129923 [interfaces.py:finish():200] Joined cpu monitor
2025-01-06 22:51:29,696 INFO    HandlerThread:2129923 [interfaces.py:finish():200] Joined disk monitor
2025-01-06 22:51:29,771 INFO    HandlerThread:2129923 [interfaces.py:finish():200] Joined gpu monitor
2025-01-06 22:51:29,771 INFO    HandlerThread:2129923 [interfaces.py:finish():200] Joined memory monitor
2025-01-06 22:51:29,771 INFO    HandlerThread:2129923 [interfaces.py:finish():200] Joined network monitor
2025-01-06 22:51:29,771 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:29,771 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-06 22:51:29,771 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 3
2025-01-06 22:51:29,771 DEBUG   SenderThread:2129923 [sender.py:send():379] send: stats
2025-01-06 22:51:29,772 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:29,772 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-06 22:51:29,772 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:29,772 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-06 22:51:29,772 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 4
2025-01-06 22:51:29,772 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:29,772 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-06 22:51:29,796 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:31,797 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:36,507 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 22:51:36,508 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:36,508 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: keepalive
2025-01-06 22:51:36,509 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:36,509 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-06 22:51:36,509 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 5
2025-01-06 22:51:36,509 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 22:51:36,509 DEBUG   SenderThread:2129923 [sender.py:send():379] send: history
2025-01-06 22:51:36,509 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: summary_record
2025-01-06 22:51:36,510 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:36,510 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-06 22:51:36,510 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:36,510 DEBUG   SenderThread:2129923 [sender.py:send():379] send: summary
2025-01-06 22:51:36,511 INFO    SenderThread:2129923 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 22:51:36,511 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:36,511 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-06 22:51:36,511 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 6
2025-01-06 22:51:36,511 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:36,511 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-06 22:51:36,511 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:36,511 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-06 22:51:36,511 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 7
2025-01-06 22:51:36,511 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 22:51:36,512 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:36,512 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-06 22:51:36,512 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:36,512 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-06 22:51:36,695 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 22:51:36,800 INFO    Thread-12 :2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:38,898 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 8
2025-01-06 22:51:38,898 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 22:51:38,898 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:38,898 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-06 22:51:38,899 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:38,899 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-06 22:51:38,899 INFO    SenderThread:2129923 [job_builder.py:build():440] Attempting to build job artifact
2025-01-06 22:51:38,901 INFO    SenderThread:2129923 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-06 22:51:38,903 INFO    SenderThread:2129923 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-06 22:51:38,917 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 9
2025-01-06 22:51:38,918 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:38,918 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-06 22:51:38,918 DEBUG   SenderThread:2129923 [sender.py:send():379] send: artifact
2025-01-06 22:51:39,117 INFO    SenderThread:2129923 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMjYzMTc3OA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMjYzMTc3OA==', 'versionIndex': 429}}}
2025-01-06 22:51:39,117 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:39,117 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-06 22:51:39,118 INFO    SenderThread:2129923 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-06 22:51:39,695 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/config.yaml config.yaml
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0 events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json wandb-summary.json
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/requirements.txt requirements.txt
2025-01-06 22:51:39,801 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-metadata.json wandb-metadata.json
2025-01-06 22:51:39,802 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log output.log
2025-01-06 22:51:39,802 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/diff.patch diff.patch
2025-01-06 22:51:39,803 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/4_eval_4_2e71831cf69946749641.mp4 media/videos/eval/4_eval_4_2e71831cf69946749641.mp4
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4 media/videos/eval/1_eval_1_75f06614e77b624267b2.mp4
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4 media/videos/eval/3_eval_3_e9e68fb8bbd70aad7917.mp4
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4 media/videos/eval/0_eval_0_4a689b02b00d8ffc5d19.mp4
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4 media/videos/eval/2_eval_2_2e12dd92f9893feab825.mp4
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json media/table/eval/total_success_rate_ci_6_d97ff5cc0ed67ab5137d.table.json
2025-01-06 22:51:39,804 INFO    SenderThread:2129923 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json media/table/eval/success_rate_ci_5_21a9f3410d4e4ea9386f.table.json
2025-01-06 22:51:39,805 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 10
2025-01-06 22:51:39,806 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 22:51:39,807 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:39,807 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-06 22:51:39,809 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:39,809 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-06 22:51:39,809 INFO    SenderThread:2129923 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 22:51:39,995 INFO    wandb-upload_1:2129923 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/wandb-summary.json
2025-01-06 22:51:39,997 INFO    wandb-upload_0:2129923 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/config.yaml
2025-01-06 22:51:40,018 INFO    wandb-upload_4:2129923 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/output.log
2025-01-06 22:51:40,035 INFO    wandb-upload_2:2129923 [upload_job.py:push():130] Uploaded file /tmp/tmpokvkb4jmwandb/pli8ax35-events.out.tfevents.1736221595.leonmkim-ROG-Strix-G15CS-G15CS.2129749.0
2025-01-06 22:51:40,045 INFO    wandb-upload_3:2129923 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/files/requirements.txt
2025-01-06 22:51:40,245 INFO    Thread-11 (_thread_body):2129923 [sender.py:transition_state():614] send defer: 11
2025-01-06 22:51:40,245 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:40,245 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-06 22:51:40,245 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:40,245 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-06 22:51:40,245 INFO    SenderThread:2129923 [file_pusher.py:join():175] waiting for file pusher
2025-01-06 22:51:40,247 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 12
2025-01-06 22:51:40,247 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:40,247 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-06 22:51:40,247 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:40,247 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-06 22:51:40,247 INFO    SenderThread:2129923 [file_stream.py:finish():601] file stream finish called
2025-01-06 22:51:40,683 INFO    SenderThread:2129923 [file_stream.py:finish():605] file stream finish is done
2025-01-06 22:51:40,683 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 13
2025-01-06 22:51:40,683 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:40,683 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-06 22:51:40,683 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:40,683 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-06 22:51:40,683 INFO    SenderThread:2129923 [sender.py:transition_state():614] send defer: 14
2025-01-06 22:51:40,683 DEBUG   SenderThread:2129923 [sender.py:send():379] send: final
2025-01-06 22:51:40,684 DEBUG   SenderThread:2129923 [sender.py:send():379] send: footer
2025-01-06 22:51:40,684 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: defer
2025-01-06 22:51:40,684 INFO    HandlerThread:2129923 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-06 22:51:40,684 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: defer
2025-01-06 22:51:40,684 INFO    SenderThread:2129923 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-06 22:51:40,684 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 22:51:40,684 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 22:51:40,685 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: server_info
2025-01-06 22:51:40,685 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: server_info
2025-01-06 22:51:40,686 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 22:51:40,686 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: get_summary
2025-01-06 22:51:40,687 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-06 22:51:40,688 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 22:51:40,745 DEBUG   SenderThread:2129923 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 22:51:40,746 INFO    MainThread:2129923 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-06 22:51:40,746 INFO    MainThread:2129923 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-06 22:51:40,746 INFO    MainThread:2129923 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-06 22:51:40,746 DEBUG   HandlerThread:2129923 [handler.py:handle_request():158] handle_request: shutdown
2025-01-06 22:51:40,747 INFO    HandlerThread:2129923 [handler.py:finish():882] shutting down handler
2025-01-06 22:51:41,687 INFO    WriterThread:2129923 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/bbomq0jb/wandb/run-20250106_224634-bbomq0jb/run-bbomq0jb.wandb
2025-01-06 22:51:41,746 INFO    SenderThread:2129923 [sender.py:finish():1615] shutting down sender
2025-01-06 22:51:41,746 INFO    SenderThread:2129923 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 22:51:41,746 INFO    SenderThread:2129923 [file_pusher.py:join():175] waiting for file pusher