2025-01-06 22:57:16,956 INFO StreamThr :2132474 [internal.py:wandb_internal():85] W&B internal server running at pid: 2132474, started at: 2025-01-06 22:57:16.956330 2025-01-06 22:57:16,957 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status 2025-01-06 22:57:16,969 INFO WriterThread:2132474 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/run-3fs5lhth.wandb 2025-01-06 22:57:16,969 DEBUG SenderThread:2132474 [sender.py:send():379] send: header 2025-01-06 22:57:16,976 DEBUG SenderThread:2132474 [sender.py:send():379] send: run 2025-01-06 22:57:17,225 INFO SenderThread:2132474 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files 2025-01-06 22:57:17,225 INFO SenderThread:2132474 [sender.py:_start_run_threads():1188] run started: 3fs5lhth with start time 1736222236.956636 2025-01-06 22:57:17,241 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: check_version 2025-01-06 22:57:17,242 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: check_version 2025-01-06 22:57:17,288 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: run_start 2025-01-06 22:57:17,293 DEBUG HandlerThread:2132474 [system_info.py:__init__():26] System info init 2025-01-06 22:57:17,293 DEBUG HandlerThread:2132474 [system_info.py:__init__():41] System info init done 2025-01-06 22:57:17,293 INFO HandlerThread:2132474 [system_monitor.py:start():194] Starting system monitor 2025-01-06 22:57:17,293 INFO SystemMonitor:2132474 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-06 22:57:17,293 INFO HandlerThread:2132474 [system_monitor.py:probe():214] Collecting system info 2025-01-06 22:57:17,293 INFO SystemMonitor:2132474 [interfaces.py:start():188] Started cpu monitoring 2025-01-06 22:57:17,294 INFO SystemMonitor:2132474 [interfaces.py:start():188] Started disk monitoring 2025-01-06 22:57:17,295 INFO SystemMonitor:2132474 [interfaces.py:start():188] Started gpu monitoring 2025-01-06 22:57:17,295 INFO SystemMonitor:2132474 [interfaces.py:start():188] Started memory monitoring 2025-01-06 22:57:17,296 INFO SystemMonitor:2132474 [interfaces.py:start():188] Started network monitoring 2025-01-06 22:57:17,306 DEBUG HandlerThread:2132474 [system_info.py:probe():152] Probing system 2025-01-06 22:57:17,307 DEBUG HandlerThread:2132474 [system_info.py:_probe_git():137] Probing git 2025-01-06 22:57:17,312 DEBUG HandlerThread:2132474 [system_info.py:_probe_git():145] Probing git done 2025-01-06 22:57:17,312 DEBUG HandlerThread:2132474 [system_info.py:probe():200] Probing system done 2025-01-06 22:57:17,312 DEBUG HandlerThread:2132474 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T03:57:17.306273', 'startedAt': '2025-01-07T03:57:16.946794', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.0555, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.908, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 638.5600509643555}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-06 22:57:17,312 INFO HandlerThread:2132474 [system_monitor.py:probe():224] Finished collecting system info 2025-01-06 22:57:17,312 INFO HandlerThread:2132474 [system_monitor.py:probe():227] Publishing system info 2025-01-06 22:57:17,312 DEBUG HandlerThread:2132474 [system_info.py:_save_code():44] Saving code 2025-01-06 22:57:17,313 DEBUG HandlerThread:2132474 [system_info.py:_save_code():67] Saving code done 2025-01-06 22:57:17,313 DEBUG HandlerThread:2132474 [system_info.py:_save_patches():84] Saving git patches 2025-01-06 22:57:17,377 DEBUG HandlerThread:2132474 [system_info.py:_save_patches():126] Saving git patches done 2025-01-06 22:57:17,377 INFO HandlerThread:2132474 [system_monitor.py:probe():229] Finished publishing system info 2025-01-06 22:57:17,380 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 22:57:17,380 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-06 22:57:17,381 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-06 22:57:17,381 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-06 22:57:17,627 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/39oobfg6-wandb-metadata.json 2025-01-06 22:57:17,629 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: python_packages 2025-01-06 22:57:17,629 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: python_packages 2025-01-06 22:57:17,629 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:57:17,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:57:17,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:17,631 INFO HandlerThread:2132474 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/tb" save: true } control { relay_id: "bc2d8352641d45d99d74edbb889444b7" } _info { stream_id: "3fs5lhth" } 2025-01-06 22:57:17,632 DEBUG HandlerThread:2132474 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-06 22:57:17,649 INFO wandb-upload_2:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/kv49xzv5-diff.patch 2025-01-06 22:57:17,652 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/639feo2f-code/FISH/eval_robot.py 2025-01-06 22:57:17,762 DEBUG SenderThread:2132474 [sender.py:send():379] send: telemetry 2025-01-06 22:57:17,763 DEBUG SenderThread:2132474 [sender.py:send():379] send: tbrecord 2025-01-06 22:57:17,763 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 22:57:17,763 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 with policy live 2025-01-06 22:57:17,951 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/hr5aukav-events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 22:57:18,226 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/requirements.txt 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/code/FISH/eval_robot.py 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-metadata.json 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/diff.patch 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/code 2025-01-06 22:57:18,227 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/code/FISH 2025-01-06 22:57:18,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:19,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:20,226 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:20,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:21,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:22,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:22,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:23,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:24,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:25,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:26,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:27,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:27,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:28,052 DEBUG SenderThread:2132474 [sender.py:send():379] send: config 2025-01-06 22:57:28,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:29,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:30,228 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:30,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:31,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:32,629 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:57:32,629 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:57:32,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:32,685 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:33,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:34,229 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:34,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:35,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:36,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:37,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:38,231 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:38,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:38,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:39,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:40,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:41,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:42,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:43,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:43,877 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:44,232 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:44,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:45,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:46,233 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:46,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:47,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:57:47,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:57:47,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:48,234 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:57:48,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:49,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:49,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:50,234 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/config.yaml 2025-01-06 22:57:50,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:51,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:52,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:53,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:54,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:55,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:57:55,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:56,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:57,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:58,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:57:59,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:00,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:00,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:01,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:02,629 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:58:02,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:58:02,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:03,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:04,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:05,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:05,747 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:06,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:07,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:08,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:09,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:10,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:11,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:11,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:12,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:13,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:14,241 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:14,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:15,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:16,241 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:16,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:17,296 DEBUG SystemMonitor:2132474 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-06 22:58:17,297 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 22:58:17,298 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:17,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:58:17,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:58:17,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:18,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:19,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:20,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:21,637 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:22,440 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:22,450 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 22:58:22,450 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 with policy now 2025-01-06 22:58:22,451 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 22:58:22,452 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 22:58:22,452 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 22:58:22,452 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 22:58:22,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:22,717 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/la2621ki-media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 2025-01-06 22:58:23,243 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 22:58:23,243 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 2025-01-06 22:58:23,243 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval 2025-01-06 22:58:23,243 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos 2025-01-06 22:58:23,244 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media 2025-01-06 22:58:23,539 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/tqbrjbop-media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 2025-01-06 22:58:23,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:24,243 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:24,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:25,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:26,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:27,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:27,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:28,245 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:28,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:29,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:30,245 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:30,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:31,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:32,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:58:32,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:58:32,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:32,790 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:33,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:34,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:35,639 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:36,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:37,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:38,467 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:38,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:39,249 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 22:58:39,490 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/13k8zo9y-events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 22:58:39,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:40,249 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:40,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:41,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:42,249 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:42,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:43,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:43,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:44,250 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:44,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:45,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:46,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:47,298 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 22:58:47,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:58:47,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:58:47,641 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:48,252 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:58:48,641 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:48,781 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:49,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:50,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:51,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:52,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:53,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:54,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:54,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:55,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:56,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:57,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:58,643 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:58:59,633 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:58:59,643 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:00,643 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:01,643 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:02,256 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:02,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:59:02,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:59:02,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:03,644 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:04,644 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:05,005 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:05,644 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:06,257 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:06,644 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:07,644 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:08,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:09,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:10,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:10,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:11,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:12,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:13,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:14,260 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:14,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:15,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:15,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:16,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:17,298 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 22:59:17,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:59:17,630 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:59:17,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:18,261 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:18,288 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 22:59:18,288 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 with policy now 2025-01-06 22:59:18,293 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 22:59:18,293 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 22:59:18,293 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 22:59:18,298 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 22:59:18,557 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/awrjwd0o-media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 2025-01-06 22:59:18,646 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:19,261 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 22:59:19,261 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 2025-01-06 22:59:19,262 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval 2025-01-06 22:59:19,601 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/ibe20cqo-media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 2025-01-06 22:59:19,648 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:20,262 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:20,651 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:21,262 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:21,319 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:21,665 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:22,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:23,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:24,263 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:24,649 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:25,649 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:26,264 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:26,437 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:26,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:27,264 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 22:59:27,649 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:28,070 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 22:59:28,070 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 22:59:28,070 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 22:59:28,264 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 22:59:28,265 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:28,649 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:29,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:30,265 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:30,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:31,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:31,981 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:32,266 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:32,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:59:32,631 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:59:32,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:33,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:34,650 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:35,651 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:36,651 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:37,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:37,651 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:38,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:39,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:40,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:41,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:42,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:42,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:43,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:44,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:45,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:46,652 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:47,299 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 22:59:47,630 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 22:59:47,631 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 22:59:47,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:47,729 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:48,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:49,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:50,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:51,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:52,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:53,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:53,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:54,653 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:55,654 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:56,274 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 22:59:56,654 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:57,655 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:58,635 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 22:59:58,655 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 22:59:59,655 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:00,655 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:01,656 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:02,631 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:00:02,631 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:00:02,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:03,656 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:03,748 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:04,656 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:05,658 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:06,662 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:06,673 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 23:00:06,673 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 with policy now 2025-01-06 23:00:06,717 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 23:00:06,718 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:00:06,718 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:00:06,728 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:00:06,944 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/8t3lkz7m-media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 2025-01-06 23:00:07,277 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:00:07,277 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 2025-01-06 23:00:07,277 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval 2025-01-06 23:00:07,565 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/3y3lkwq7-media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 2025-01-06 23:00:07,662 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:08,277 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:08,662 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:09,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:09,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:10,667 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:11,669 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:12,279 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:12,667 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:13,665 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:14,279 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:14,666 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:15,237 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:15,670 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:16,670 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:17,299 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 23:00:17,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:00:17,638 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:00:17,681 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:17,901 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:00:17,901 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:00:17,901 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:00:18,281 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:00:18,281 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:00:18,545 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/cheei9vj-events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:00:18,670 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:19,672 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:20,281 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:20,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:20,741 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:21,672 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:22,282 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:22,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:23,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:24,283 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:24,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:25,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:25,838 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:26,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:27,671 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:28,672 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:29,672 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:30,285 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:30,672 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:31,317 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:31,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:32,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:00:32,632 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:00:32,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:33,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:34,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:35,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:36,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:36,717 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:37,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:38,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:39,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:40,673 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:41,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:42,638 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:42,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:43,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:44,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:45,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:46,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:47,300 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 23:00:47,632 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:00:47,633 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:00:47,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:47,789 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:48,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:49,674 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:50,675 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:51,676 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:52,292 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:52,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:53,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:53,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:54,676 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:55,676 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:56,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:57,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:58,295 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:00:58,510 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 23:00:58,510 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 with policy now 2025-01-06 23:00:58,515 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 23:00:58,515 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:00:58,516 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:00:58,527 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:00:58,677 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:00:58,767 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/ztlqoqfp-media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 2025-01-06 23:00:59,295 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:00:59,295 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 2025-01-06 23:00:59,295 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval 2025-01-06 23:00:59,553 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:00:59,582 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/q4b0n1a9-media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 2025-01-06 23:00:59,683 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:00,295 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:00,679 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:01,678 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:02,640 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:01:02,640 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:01:02,685 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:03,684 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:04,297 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:04,686 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:04,811 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:05,688 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:06,297 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:06,689 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:07,699 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:08,698 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:09,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:10,645 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:10,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:11,699 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:12,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:13,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:14,300 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:01:14,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:14,966 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:01:14,966 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:01:14,967 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:01:15,300 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:01:15,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:16,301 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:16,313 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:16,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:17,300 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 23:01:17,636 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:01:17,636 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:01:17,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:18,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:19,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:20,302 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:20,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:21,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:22,275 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:22,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:23,701 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:24,304 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:24,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:25,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:26,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:27,641 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:27,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:28,702 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:29,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:30,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:31,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:32,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:01:32,634 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:01:32,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:32,758 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:33,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:34,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:35,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:36,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:37,703 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:38,641 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:38,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:39,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:40,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:41,704 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:42,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:43,642 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:43,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:44,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:45,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:46,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:47,300 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 23:01:47,634 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:01:47,634 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:01:47,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:48,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:48,794 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:49,706 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:50,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:51,705 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:52,706 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:53,722 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:54,209 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:54,314 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:54,354 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 23:01:54,355 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 with policy now 2025-01-06 23:01:54,364 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 23:01:54,365 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:01:54,365 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:01:54,371 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:01:54,618 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/8isq0r7b-media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 2025-01-06 23:01:54,725 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:55,315 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:01:55,315 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 2025-01-06 23:01:55,315 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval 2025-01-06 23:01:55,600 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/ql40hy80-media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 2025-01-06 23:01:55,729 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:56,315 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:56,735 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:57,315 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:01:57,729 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:58,724 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:01:59,658 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:01:59,722 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:00,316 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:00,733 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:01,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:02,317 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:02,648 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 23:02:02,648 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: stop_status 2025-01-06 23:02:02,777 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:03,025 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:02:03,026 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:03,026 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:03,202 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 23:02:03,202 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 23:02:03,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:03,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:02:03,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-06 23:02:03,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media 2025-01-06 23:02:03,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval 2025-01-06 23:02:03,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table 2025-01-06 23:02:03,434 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 23:02:03,519 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/1w9ag9fk-media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-06 23:02:03,554 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/pcgi9dpu-events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:02:03,680 INFO wandb-upload_2:2132474 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphl6xy5fo 2025-01-06 23:02:03,725 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:03,852 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 23:02:03,863 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 23:02:04,077 INFO SenderThread:2132474 [sender.py:send_request_log_artifact():1525] logged artifact run-3fs5lhth-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjY2OTc1OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc3MTAxMA==', 'latestArtifact': None}} 2025-01-06 23:02:04,077 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 23:02:04,077 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json with policy now 2025-01-06 23:02:04,078 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:02:04,078 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,079 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,079 DEBUG SenderThread:2132474 [sender.py:send():379] send: telemetry 2025-01-06 23:02:04,079 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,079 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,080 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,080 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,080 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,080 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,081 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,081 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,081 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,082 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,083 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,083 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,083 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,084 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,084 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,084 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,084 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,085 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,085 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,085 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,085 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,086 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:04,086 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 23:02:04,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:04,318 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:02:04,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:04,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json 2025-01-06 23:02:04,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval 2025-01-06 23:02:04,327 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/jtm3phv8-media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-06 23:02:04,483 INFO wandb-upload_2:2132474 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpyd6j289m 2025-01-06 23:02:04,499 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/0q3c0k8h-media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json 2025-01-06 23:02:04,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:04,865 INFO SenderThread:2132474 [sender.py:send_request_log_artifact():1525] logged artifact run-3fs5lhth-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjY2OTc5OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc3MTAxNQ==', 'latestArtifact': None}} 2025-01-06 23:02:04,865 DEBUG SenderThread:2132474 [sender.py:send():379] send: files 2025-01-06 23:02:04,865 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json with policy now 2025-01-06 23:02:04,866 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:02:04,866 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:02:04,866 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:04,867 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:05,134 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/qn8om7sk-media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json 2025-01-06 23:02:05,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:05,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:06,319 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:06,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:07,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:08,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:09,726 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:10,649 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:02:10,727 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:11,321 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/config.yaml 2025-01-06 23:02:11,727 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:12,731 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:13,727 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:13,865 DEBUG SenderThread:2132474 [sender.py:send():379] send: exit 2025-01-06 23:02:13,865 INFO SenderThread:2132474 [sender.py:send_exit():586] handling exit code: 0 2025-01-06 23:02:13,866 INFO SenderThread:2132474 [sender.py:send_exit():588] handling runtime: 296 2025-01-06 23:02:13,867 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:13,867 INFO SenderThread:2132474 [sender.py:send_exit():594] send defer 2025-01-06 23:02:13,867 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:13,867 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-06 23:02:13,867 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:13,867 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-06 23:02:13,867 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 1 2025-01-06 23:02:13,867 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:13,867 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-06 23:02:13,867 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:13,868 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-06 23:02:13,868 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 2 2025-01-06 23:02:13,868 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:13,868 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-06 23:02:13,868 INFO HandlerThread:2132474 [system_monitor.py:finish():203] Stopping system monitor 2025-01-06 23:02:13,868 DEBUG SystemMonitor:2132474 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-06 23:02:13,868 DEBUG SystemMonitor:2132474 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-06 23:02:13,869 INFO HandlerThread:2132474 [interfaces.py:finish():200] Joined cpu monitor 2025-01-06 23:02:13,869 INFO HandlerThread:2132474 [interfaces.py:finish():200] Joined disk monitor 2025-01-06 23:02:13,920 INFO HandlerThread:2132474 [interfaces.py:finish():200] Joined gpu monitor 2025-01-06 23:02:13,920 INFO HandlerThread:2132474 [interfaces.py:finish():200] Joined memory monitor 2025-01-06 23:02:13,921 INFO HandlerThread:2132474 [interfaces.py:finish():200] Joined network monitor 2025-01-06 23:02:13,921 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:13,921 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-06 23:02:13,921 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 3 2025-01-06 23:02:13,921 DEBUG SenderThread:2132474 [sender.py:send():379] send: stats 2025-01-06 23:02:13,921 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:13,921 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-06 23:02:13,921 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:13,922 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-06 23:02:13,922 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 4 2025-01-06 23:02:13,922 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:13,922 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-06 23:02:14,322 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:16,323 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:20,032 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 23:02:20,032 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:02:20,032 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: keepalive 2025-01-06 23:02:20,034 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,034 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-06 23:02:20,034 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 5 2025-01-06 23:02:20,034 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 23:02:20,034 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:20,034 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-06 23:02:20,034 DEBUG SenderThread:2132474 [sender.py:send():379] send: history 2025-01-06 23:02:20,035 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: summary_record 2025-01-06 23:02:20,035 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:20,035 DEBUG SenderThread:2132474 [sender.py:send():379] send: summary 2025-01-06 23:02:20,036 INFO SenderThread:2132474 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 23:02:20,036 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,036 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-06 23:02:20,036 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 6 2025-01-06 23:02:20,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:20,036 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-06 23:02:20,036 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,036 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-06 23:02:20,036 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 7 2025-01-06 23:02:20,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 23:02:20,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:20,036 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-06 23:02:20,037 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,037 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-06 23:02:20,325 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:20,487 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 8 2025-01-06 23:02:20,487 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:20,488 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-06 23:02:20,488 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,488 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-06 23:02:20,488 INFO SenderThread:2132474 [job_builder.py:build():440] Attempting to build job artifact 2025-01-06 23:02:20,490 INFO SenderThread:2132474 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-06 23:02:20,492 INFO SenderThread:2132474 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-06 23:02:20,506 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 9 2025-01-06 23:02:20,506 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:20,506 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-06 23:02:20,507 DEBUG SenderThread:2132474 [sender.py:send():379] send: artifact 2025-01-06 23:02:20,734 INFO SenderThread:2132474 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMjY1NzQxMQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMjY1NzQxMQ==', 'versionIndex': 430}}} 2025-01-06 23:02:20,734 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:20,734 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-06 23:02:20,734 INFO SenderThread:2132474 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-06 23:02:20,867 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 23:02:21,325 INFO Thread-12 :2132474 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:21,326 INFO SenderThread:2132474 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files 2025-01-06 23:02:21,326 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/config.yaml config.yaml 2025-01-06 23:02:21,326 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json wandb-summary.json 2025-01-06 23:02:21,328 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/requirements.txt requirements.txt 2025-01-06 23:02:21,328 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-metadata.json wandb-metadata.json 2025-01-06 23:02:21,328 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:02:21,330 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log output.log 2025-01-06 23:02:21,330 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/diff.patch diff.patch 2025-01-06 23:02:21,331 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 media/videos/eval/3_eval_3_ebb2a03c3b35f1d33435.mp4 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 media/videos/eval/1_eval_1_487641939ef48b883b67.mp4 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 media/videos/eval/4_eval_4_e244931479443b8b04bf.mp4 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 media/videos/eval/2_eval_2_e00f3422b92e36dcbe33.mp4 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 media/videos/eval/0_eval_0_0b48560eafc6e2046c0d.mp4 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json media/table/eval/total_success_rate_ci_6_a90222fd6986d0de60ca.table.json 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-06 23:02:21,332 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 10 2025-01-06 23:02:21,332 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 23:02:21,333 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:21,334 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-06 23:02:21,335 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:21,336 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-06 23:02:21,336 INFO SenderThread:2132474 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 23:02:21,493 INFO wandb-upload_2:2132474 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/requirements.txt 2025-01-06 23:02:21,499 INFO wandb-upload_0:2132474 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/wandb-summary.json 2025-01-06 23:02:21,548 INFO wandb-upload_1:2132474 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/config.yaml 2025-01-06 23:02:21,583 INFO wandb-upload_4:2132474 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/files/output.log 2025-01-06 23:02:21,608 INFO wandb-upload_3:2132474 [upload_job.py:push():130] Uploaded file /tmp/tmpg5vamc1mwandb/pqyy08m5-events.out.tfevents.1736222237.leonmkim-ROG-Strix-G15CS-G15CS.2132300.0 2025-01-06 23:02:21,809 INFO Thread-11 (_thread_body):2132474 [sender.py:transition_state():614] send defer: 11 2025-01-06 23:02:21,809 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:21,809 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-06 23:02:21,809 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:21,809 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-06 23:02:21,809 INFO SenderThread:2132474 [file_pusher.py:join():175] waiting for file pusher 2025-01-06 23:02:21,811 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 12 2025-01-06 23:02:21,811 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:21,811 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-06 23:02:21,811 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:21,811 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-06 23:02:21,811 INFO SenderThread:2132474 [file_stream.py:finish():601] file stream finish called 2025-01-06 23:02:21,867 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 23:02:22,034 INFO SenderThread:2132474 [file_stream.py:finish():605] file stream finish is done 2025-01-06 23:02:22,034 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 13 2025-01-06 23:02:22,034 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 23:02:22,034 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:22,034 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-06 23:02:22,034 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:22,034 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-06 23:02:22,035 INFO SenderThread:2132474 [sender.py:transition_state():614] send defer: 14 2025-01-06 23:02:22,035 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: defer 2025-01-06 23:02:22,035 INFO HandlerThread:2132474 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-06 23:02:22,035 DEBUG SenderThread:2132474 [sender.py:send():379] send: final 2025-01-06 23:02:22,035 DEBUG SenderThread:2132474 [sender.py:send():379] send: footer 2025-01-06 23:02:22,035 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: defer 2025-01-06 23:02:22,035 INFO SenderThread:2132474 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-06 23:02:22,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 23:02:22,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: server_info 2025-01-06 23:02:22,036 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 23:02:22,036 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 23:02:22,036 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: server_info 2025-01-06 23:02:22,037 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: get_summary 2025-01-06 23:02:22,038 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-06 23:02:22,039 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 23:02:22,098 DEBUG SenderThread:2132474 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 23:02:22,098 INFO MainThread:2132474 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-06 23:02:22,098 INFO MainThread:2132474 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-06 23:02:22,099 INFO MainThread:2132474 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-06 23:02:22,099 DEBUG HandlerThread:2132474 [handler.py:handle_request():158] handle_request: shutdown 2025-01-06 23:02:22,099 INFO HandlerThread:2132474 [handler.py:finish():882] shutting down handler 2025-01-06 23:02:23,037 INFO WriterThread:2132474 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1017_0/3fs5lhth/wandb/run-20250106_225716-3fs5lhth/run-3fs5lhth.wandb 2025-01-06 23:02:23,098 INFO SenderThread:2132474 [sender.py:finish():1615] shutting down sender 2025-01-06 23:02:23,098 INFO SenderThread:2132474 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 23:02:23,098 INFO SenderThread:2132474 [file_pusher.py:join():175] waiting for file pusher