2025-01-06 21:00:53,097 INFO StreamThr :2117937 [internal.py:wandb_internal():85] W&B internal server running at pid: 2117937, started at: 2025-01-06 21:00:53.097352 2025-01-06 21:00:53,099 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status 2025-01-06 21:00:53,111 INFO WriterThread:2117937 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/run-3d3i07nt.wandb 2025-01-06 21:00:53,112 DEBUG SenderThread:2117937 [sender.py:send():379] send: header 2025-01-06 21:00:53,120 DEBUG SenderThread:2117937 [sender.py:send():379] send: run 2025-01-06 21:00:53,382 INFO SenderThread:2117937 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files 2025-01-06 21:00:53,382 INFO SenderThread:2117937 [sender.py:_start_run_threads():1188] run started: 3d3i07nt with start time 1736215253.098176 2025-01-06 21:00:53,401 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: check_version 2025-01-06 21:00:53,401 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: check_version 2025-01-06 21:00:53,489 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: run_start 2025-01-06 21:00:53,494 DEBUG HandlerThread:2117937 [system_info.py:__init__():26] System info init 2025-01-06 21:00:53,494 DEBUG HandlerThread:2117937 [system_info.py:__init__():41] System info init done 2025-01-06 21:00:53,494 INFO HandlerThread:2117937 [system_monitor.py:start():194] Starting system monitor 2025-01-06 21:00:53,494 INFO SystemMonitor:2117937 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-06 21:00:53,494 INFO HandlerThread:2117937 [system_monitor.py:probe():214] Collecting system info 2025-01-06 21:00:53,495 INFO SystemMonitor:2117937 [interfaces.py:start():188] Started cpu monitoring 2025-01-06 21:00:53,495 INFO SystemMonitor:2117937 [interfaces.py:start():188] Started disk monitoring 2025-01-06 21:00:53,496 INFO SystemMonitor:2117937 [interfaces.py:start():188] Started gpu monitoring 2025-01-06 21:00:53,496 INFO SystemMonitor:2117937 [interfaces.py:start():188] Started memory monitoring 2025-01-06 21:00:53,497 INFO SystemMonitor:2117937 [interfaces.py:start():188] Started network monitoring 2025-01-06 21:00:53,507 DEBUG HandlerThread:2117937 [system_info.py:probe():152] Probing system 2025-01-06 21:00:53,509 DEBUG HandlerThread:2117937 [system_info.py:_probe_git():137] Probing git 2025-01-06 21:00:53,513 DEBUG HandlerThread:2117937 [system_info.py:_probe_git():145] Probing git done 2025-01-06 21:00:53,513 DEBUG HandlerThread:2117937 [system_info.py:probe():200] Probing system done 2025-01-06 21:00:53,514 DEBUG HandlerThread:2117937 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T02:00:53.507771', 'startedAt': '2025-01-07T02:00:53.088016', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.778875, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.019, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 538.4776840209961}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-06 21:00:53,514 INFO HandlerThread:2117937 [system_monitor.py:probe():224] Finished collecting system info 2025-01-06 21:00:53,514 INFO HandlerThread:2117937 [system_monitor.py:probe():227] Publishing system info 2025-01-06 21:00:53,514 DEBUG HandlerThread:2117937 [system_info.py:_save_code():44] Saving code 2025-01-06 21:00:53,515 DEBUG HandlerThread:2117937 [system_info.py:_save_code():67] Saving code done 2025-01-06 21:00:53,515 DEBUG HandlerThread:2117937 [system_info.py:_save_patches():84] Saving git patches 2025-01-06 21:00:53,584 DEBUG HandlerThread:2117937 [system_info.py:_save_patches():126] Saving git patches done 2025-01-06 21:00:53,584 INFO HandlerThread:2117937 [system_monitor.py:probe():229] Finished publishing system info 2025-01-06 21:00:53,587 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:00:53,587 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-06 21:00:53,587 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-06 21:00:53,588 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-06 21:00:53,840 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/cwxs2uff-code/FISH/eval_robot.py 2025-01-06 21:00:53,851 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: python_packages 2025-01-06 21:00:53,851 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: python_packages 2025-01-06 21:00:53,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:53,853 DEBUG SenderThread:2117937 [sender.py:send():379] send: telemetry 2025-01-06 21:00:53,853 INFO wandb-upload_0:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/e5jwekfy-wandb-metadata.json 2025-01-06 21:00:53,875 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/7qu68wlj-diff.patch 2025-01-06 21:00:53,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:00:53,893 INFO HandlerThread:2117937 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/tb" save: true } control { relay_id: "ced5b79432f94c96ac79791dbb41a3be" } _info { stream_id: "3d3i07nt" } 2025-01-06 21:00:53,894 DEBUG HandlerThread:2117937 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-06 21:00:53,894 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:00:54,013 DEBUG SenderThread:2117937 [sender.py:send():379] send: tbrecord 2025-01-06 21:00:54,014 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:00:54,014 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 with policy live 2025-01-06 21:00:54,205 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/dijz3mrs-events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:00:54,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/code/FISH/eval_robot.py 2025-01-06 21:00:54,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/requirements.txt 2025-01-06 21:00:54,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-metadata.json 2025-01-06 21:00:54,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:00:54,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/diff.patch 2025-01-06 21:00:54,385 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:00:54,385 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/code/FISH 2025-01-06 21:00:54,385 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/code 2025-01-06 21:00:54,851 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:55,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:56,383 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:00:56,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:57,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:58,766 DEBUG SenderThread:2117937 [sender.py:send():379] send: config 2025-01-06 21:00:58,766 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:00:58,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:00:59,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:00,384 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:00,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:01,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:02,385 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:02,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:03,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:03,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:04,385 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:04,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:05,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:06,386 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:06,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:07,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:08,387 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:08,851 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:01:08,851 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:01:08,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:08,992 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:09,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:10,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:11,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:12,388 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:12,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:13,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:14,389 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:14,579 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:14,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:15,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:16,389 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:16,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:17,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:18,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:19,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:19,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:20,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:21,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:22,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:23,851 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:01:23,851 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:01:23,897 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:24,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:25,015 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:25,392 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/config.yaml 2025-01-06 21:01:25,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:26,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:27,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:28,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:29,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:30,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:30,856 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:31,856 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:32,856 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:33,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:34,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:35,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:36,395 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:36,685 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:36,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:37,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:38,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:01:38,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:01:38,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:39,858 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:40,396 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:40,858 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:41,858 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:42,319 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:42,530 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:01:42,531 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 with policy now 2025-01-06 21:01:42,531 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:01:42,532 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:01:42,532 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:01:42,537 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:01:42,800 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/inojqhlt-media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 2025-01-06 21:01:42,858 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:43,397 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 2025-01-06 21:01:43,398 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:01:43,398 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos 2025-01-06 21:01:43,398 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media 2025-01-06 21:01:43,398 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval 2025-01-06 21:01:43,686 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/xd62vwum-media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 2025-01-06 21:01:43,858 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:44,398 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:44,859 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:45,859 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:46,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:47,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:48,185 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:48,399 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:48,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:49,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:50,400 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:50,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:51,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:52,401 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:01:52,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:53,497 DEBUG SystemMonitor:2117937 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-06 21:01:53,498 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:01:53,498 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:53,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:01:53,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:01:53,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:54,401 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:54,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:55,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:56,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:57,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:58,403 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:01:58,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:01:58,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:01:59,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:00,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:01,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:02,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:03,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:03,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:04,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:05,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:06,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:07,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:08,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:02:08,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:02:08,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:08,918 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:09,863 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:10,863 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:11,863 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:12,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:13,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:14,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:14,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:15,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:16,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:17,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:18,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:19,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:19,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:20,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:21,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:22,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:23,507 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:02:23,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:02:23,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:02:23,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:24,179 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:02:24,179 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 with policy now 2025-01-06 21:02:24,182 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:02:24,183 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:02:24,183 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:02:24,184 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:02:24,411 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:02:24,411 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:24,411 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 2025-01-06 21:02:24,411 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval 2025-01-06 21:02:24,448 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/dnchds9h-media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 2025-01-06 21:02:24,720 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/fbs9frno-media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 2025-01-06 21:02:24,868 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:25,201 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:25,871 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:26,411 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:26,868 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:27,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:28,875 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:29,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:30,413 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:30,448 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:30,871 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:31,871 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:32,413 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:02:32,414 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:32,675 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/zupl7yxz-events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:02:32,871 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:33,372 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:02:33,372 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:02:33,373 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:02:33,414 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:02:33,872 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:34,414 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:34,872 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:35,825 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:35,871 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:36,872 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:37,872 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:38,415 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:02:38,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:02:38,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:02:38,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:39,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:40,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:41,009 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:41,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:42,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:43,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:44,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:45,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:46,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:46,873 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:47,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:48,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:49,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:50,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:51,855 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:51,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:52,874 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:53,499 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:02:53,852 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:02:53,852 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:02:53,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:54,875 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:55,875 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:56,875 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:56,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:02:57,875 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:58,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:02:59,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:00,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:01,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:02,778 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:02,887 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:03,882 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:04,249 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:03:04,249 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 with policy now 2025-01-06 21:03:04,251 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:03:04,252 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:03:04,252 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:03:04,252 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:03:04,424 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:03:04,424 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:04,424 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 2025-01-06 21:03:04,424 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval 2025-01-06 21:03:04,503 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/nnw7v8qe-media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 2025-01-06 21:03:04,796 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/cr1k5tmp-media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 2025-01-06 21:03:04,879 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:05,884 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:06,425 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:06,888 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:07,889 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:08,425 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:08,621 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:08,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:03:08,854 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:03:08,897 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:09,894 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:10,884 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:11,887 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:12,427 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:12,887 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:13,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:13,885 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:14,886 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:15,888 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:16,891 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:17,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:18,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:18,888 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:19,888 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:20,887 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:21,891 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:22,888 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:23,499 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:03:23,856 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:03:23,856 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:03:23,897 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:23,973 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:24,899 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:25,894 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:26,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:27,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:28,890 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:29,865 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:29,892 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:30,897 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:31,893 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:32,894 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:33,895 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:34,894 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:35,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:35,903 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:36,903 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:37,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:38,861 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:03:38,861 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:03:38,909 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:39,899 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:40,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:41,147 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:03:41,147 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:03:41,148 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:41,148 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:03:41,436 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:03:41,437 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:03:41,680 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/uj9tdk34-events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:03:41,903 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:42,437 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:42,905 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:43,905 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:44,437 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:44,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:45,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:46,438 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:46,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:47,004 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:47,902 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:48,901 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:49,905 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:50,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:51,905 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:52,862 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:52,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:53,500 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:03:53,853 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:03:53,854 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:03:53,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:54,441 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:03:54,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:55,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:56,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:57,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:57,995 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:03:58,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:03:59,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:00,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:01,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:02,906 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:03,862 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:03,907 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:04,907 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:05,907 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:06,908 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:07,908 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:08,854 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:04:08,854 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:04:08,908 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:08,958 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:09,909 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:10,446 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:10,910 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:11,913 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:12,447 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:12,917 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:13,230 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:04:13,230 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 with policy now 2025-01-06 21:04:13,243 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:04:13,243 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:04:13,243 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:04:13,244 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:04:13,448 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:04:13,448 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 2025-01-06 21:04:13,448 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval 2025-01-06 21:04:13,492 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/8e25x4z2-media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 2025-01-06 21:04:13,760 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/wy76zwe0-media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 2025-01-06 21:04:13,923 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:14,448 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:14,866 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:14,917 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:15,921 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:16,921 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:17,924 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:18,449 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:18,922 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:19,921 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:20,409 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:20,450 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:20,928 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:21,927 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:22,924 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:23,501 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:04:23,860 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:04:23,860 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:04:23,922 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:24,200 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:04:24,200 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:04:24,201 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:04:24,451 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:04:24,452 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:04:24,930 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:25,929 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:26,117 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:26,452 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:26,938 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:27,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:28,940 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:29,931 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:30,454 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:30,931 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:31,591 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:31,931 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:32,931 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:33,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:34,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:35,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:36,867 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:36,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:37,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:38,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:04:38,857 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:04:38,932 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:39,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:40,457 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:40,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:41,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:41,978 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:42,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:43,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:44,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:45,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:46,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:47,867 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:47,933 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:48,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:49,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:50,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:51,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:52,870 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:52,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:53,501 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:04:53,857 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:04:53,857 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:04:53,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:54,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:55,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:56,939 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:57,669 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:04:57,669 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 with policy now 2025-01-06 21:04:57,674 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:04:57,674 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:04:57,674 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:04:57,675 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:04:57,937 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/w0jcvthq-media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 2025-01-06 21:04:57,938 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:58,478 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:04:58,478 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:04:58,478 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 2025-01-06 21:04:58,479 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval 2025-01-06 21:04:58,706 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:04:58,770 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/05y7ayc2-media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 2025-01-06 21:04:58,952 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:04:59,943 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:00,949 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:01,953 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:02,479 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:02,952 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:03,876 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:03,958 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:04,953 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:05,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:06,481 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:06,956 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:07,952 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:08,864 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:05:08,864 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:05:08,951 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:08,989 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:09,957 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:10,956 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:11,959 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:12,951 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:13,953 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:14,878 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:14,959 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:15,956 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:16,955 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:17,958 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:18,970 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:19,882 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:19,965 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:20,968 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:21,965 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:22,969 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:23,501 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:05:23,870 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 21:05:23,871 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: stop_status 2025-01-06 21:05:23,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:23,998 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 21:05:24,011 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 21:05:24,268 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:05:24,474 INFO wandb-upload_2:2117937 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp7ygkp7hl 2025-01-06 21:05:24,487 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:05:24,487 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:24,487 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2025-01-06 21:05:24,487 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media 2025-01-06 21:05:24,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval 2025-01-06 21:05:24,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table 2025-01-06 21:05:24,679 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 21:05:24,687 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 21:05:24,959 INFO SenderThread:2117937 [sender.py:send_request_log_artifact():1525] logged artifact run-3d3i07nt-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjM3MDQwNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY2NTMxOQ==', 'latestArtifact': None}} 2025-01-06 21:05:24,959 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:05:24,961 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:24,961 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json with policy now 2025-01-06 21:05:24,961 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:05:24,961 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,962 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,962 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:05:24,962 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:24,963 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,963 DEBUG SenderThread:2117937 [sender.py:send():379] send: telemetry 2025-01-06 21:05:24,963 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,963 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,963 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,964 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,964 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,964 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,964 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,964 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,965 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,965 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,965 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,965 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,966 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,966 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,966 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,966 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,966 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,967 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,967 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,967 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,967 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:24,968 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:24,968 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 21:05:25,205 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/nesakpao-events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:05:25,210 INFO wandb-upload_0:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/fadlkzzd-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2025-01-06 21:05:25,369 INFO wandb-upload_2:2117937 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpwont73sm 2025-01-06 21:05:25,449 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/jmvf3due-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2025-01-06 21:05:25,487 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:05:25,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:05:25,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json 2025-01-06 21:05:25,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval 2025-01-06 21:05:25,733 INFO SenderThread:2117937 [sender.py:send_request_log_artifact():1525] logged artifact run-3d3i07nt-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjM3MDU1Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY2NTQwMQ==', 'latestArtifact': None}} 2025-01-06 21:05:25,733 DEBUG SenderThread:2117937 [sender.py:send():379] send: files 2025-01-06 21:05:25,733 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json with policy now 2025-01-06 21:05:25,733 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:05:25,734 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:25,734 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:25,908 INFO wandb-upload_0:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/g94hhcqb-media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json 2025-01-06 21:05:25,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:26,157 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/h17ehwwa-media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json 2025-01-06 21:05:26,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:05:26,488 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:26,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:27,963 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:28,963 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:29,962 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:30,005 DEBUG SenderThread:2117937 [sender.py:send():379] send: exit 2025-01-06 21:05:30,005 INFO SenderThread:2117937 [sender.py:send_exit():586] handling exit code: 0 2025-01-06 21:05:30,006 INFO SenderThread:2117937 [sender.py:send_exit():588] handling runtime: 276 2025-01-06 21:05:30,006 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:30,006 INFO SenderThread:2117937 [sender.py:send_exit():594] send defer 2025-01-06 21:05:30,006 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:30,006 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-06 21:05:30,006 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:30,006 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-06 21:05:30,006 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 1 2025-01-06 21:05:30,007 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:30,007 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-06 21:05:30,007 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:30,007 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-06 21:05:30,007 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 2 2025-01-06 21:05:30,007 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:30,007 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-06 21:05:30,007 INFO HandlerThread:2117937 [system_monitor.py:finish():203] Stopping system monitor 2025-01-06 21:05:30,007 DEBUG SystemMonitor:2117937 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-06 21:05:30,008 DEBUG SystemMonitor:2117937 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-06 21:05:30,007 INFO HandlerThread:2117937 [interfaces.py:finish():200] Joined cpu monitor 2025-01-06 21:05:30,008 INFO HandlerThread:2117937 [interfaces.py:finish():200] Joined disk monitor 2025-01-06 21:05:30,059 INFO HandlerThread:2117937 [interfaces.py:finish():200] Joined gpu monitor 2025-01-06 21:05:30,059 INFO HandlerThread:2117937 [interfaces.py:finish():200] Joined memory monitor 2025-01-06 21:05:30,059 INFO HandlerThread:2117937 [interfaces.py:finish():200] Joined network monitor 2025-01-06 21:05:30,060 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:30,060 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-06 21:05:30,060 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 3 2025-01-06 21:05:30,060 DEBUG SenderThread:2117937 [sender.py:send():379] send: stats 2025-01-06 21:05:30,060 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:30,060 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-06 21:05:30,060 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:30,060 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-06 21:05:30,060 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 4 2025-01-06 21:05:30,060 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:30,061 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-06 21:05:30,489 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:05:32,490 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:36,277 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 21:05:36,278 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:36,278 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-06 21:05:36,278 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 5 2025-01-06 21:05:36,278 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:36,278 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 21:05:36,278 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: keepalive 2025-01-06 21:05:36,279 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:36,279 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-06 21:05:36,280 DEBUG SenderThread:2117937 [sender.py:send():379] send: history 2025-01-06 21:05:36,280 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: summary_record 2025-01-06 21:05:36,281 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:36,281 DEBUG SenderThread:2117937 [sender.py:send():379] send: summary 2025-01-06 21:05:36,285 INFO SenderThread:2117937 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 21:05:36,285 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:36,285 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-06 21:05:36,286 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 6 2025-01-06 21:05:36,286 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:36,286 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-06 21:05:36,286 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:36,286 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-06 21:05:36,289 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 21:05:36,480 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 7 2025-01-06 21:05:36,481 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:36,481 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-06 21:05:36,481 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:36,481 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-06 21:05:36,492 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/config.yaml 2025-01-06 21:05:36,492 INFO Thread-12 :2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:05:37,007 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 21:05:40,281 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 8 2025-01-06 21:05:40,281 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 21:05:40,281 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:40,282 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-06 21:05:40,282 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:40,282 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-06 21:05:40,282 INFO SenderThread:2117937 [job_builder.py:build():440] Attempting to build job artifact 2025-01-06 21:05:40,284 INFO SenderThread:2117937 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-06 21:05:40,286 INFO SenderThread:2117937 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-06 21:05:40,300 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 9 2025-01-06 21:05:40,300 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:40,301 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-06 21:05:40,301 DEBUG SenderThread:2117937 [sender.py:send():379] send: artifact 2025-01-06 21:05:40,489 INFO SenderThread:2117937 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjU3MQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} 2025-01-06 21:05:40,489 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:40,489 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-06 21:05:40,489 INFO SenderThread:2117937 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-06 21:05:40,493 INFO SenderThread:2117937 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:40,493 INFO SenderThread:2117937 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/config.yaml config.yaml 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json wandb-summary.json 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/requirements.txt requirements.txt 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-metadata.json wandb-metadata.json 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log output.log 2025-01-06 21:05:40,494 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/diff.patch diff.patch 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 media/videos/eval/0_eval_0_3bab73a244800e000493.mp4 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 media/videos/eval/4_eval_4_d51249e60c287ad366a8.mp4 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 media/videos/eval/2_eval_2_559e440a12c1c69e9e1a.mp4 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 media/videos/eval/3_eval_3_9a07183c07b2b4777782.mp4 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 media/videos/eval/1_eval_1_9cbaf6111e2b85cffc2c.mp4 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json media/table/eval/total_success_rate_ci_6_b2a679debf83a2b3c3cf.table.json 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2025-01-06 21:05:40,498 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 10 2025-01-06 21:05:40,499 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:40,503 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-06 21:05:40,504 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:40,505 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-06 21:05:40,505 INFO SenderThread:2117937 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 21:05:40,675 INFO wandb-upload_0:2117937 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/config.yaml 2025-01-06 21:05:40,707 INFO wandb-upload_2:2117937 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/wandb-summary.json 2025-01-06 21:05:40,728 INFO wandb-upload_1:2117937 [upload_job.py:push():130] Uploaded file /tmp/tmp8zvqyydjwandb/mnxincra-events.out.tfevents.1736215253.leonmkim-ROG-Strix-G15CS-G15CS.2117773.0 2025-01-06 21:05:40,731 INFO wandb-upload_4:2117937 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/output.log 2025-01-06 21:05:40,733 INFO wandb-upload_3:2117937 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/files/requirements.txt 2025-01-06 21:05:40,934 INFO Thread-11 (_thread_body):2117937 [sender.py:transition_state():614] send defer: 11 2025-01-06 21:05:40,934 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:40,934 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-06 21:05:40,934 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:40,935 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-06 21:05:40,935 INFO SenderThread:2117937 [file_pusher.py:join():175] waiting for file pusher 2025-01-06 21:05:40,936 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 12 2025-01-06 21:05:40,936 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:40,936 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-06 21:05:40,936 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:40,936 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-06 21:05:40,936 INFO SenderThread:2117937 [file_stream.py:finish():601] file stream finish called 2025-01-06 21:05:41,008 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 21:05:41,162 INFO SenderThread:2117937 [file_stream.py:finish():605] file stream finish is done 2025-01-06 21:05:41,162 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 13 2025-01-06 21:05:41,162 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 21:05:41,162 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:41,162 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-06 21:05:41,162 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:41,162 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-06 21:05:41,162 INFO SenderThread:2117937 [sender.py:transition_state():614] send defer: 14 2025-01-06 21:05:41,162 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: defer 2025-01-06 21:05:41,162 INFO HandlerThread:2117937 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-06 21:05:41,163 DEBUG SenderThread:2117937 [sender.py:send():379] send: final 2025-01-06 21:05:41,163 DEBUG SenderThread:2117937 [sender.py:send():379] send: footer 2025-01-06 21:05:41,163 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: defer 2025-01-06 21:05:41,163 INFO SenderThread:2117937 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-06 21:05:41,163 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 21:05:41,164 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: server_info 2025-01-06 21:05:41,164 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 21:05:41,164 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: server_info 2025-01-06 21:05:41,166 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 21:05:41,166 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: get_summary 2025-01-06 21:05:41,167 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-06 21:05:41,167 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 21:05:41,229 DEBUG SenderThread:2117937 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 21:05:41,230 INFO MainThread:2117937 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-06 21:05:41,230 INFO MainThread:2117937 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-06 21:05:41,230 INFO MainThread:2117937 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-06 21:05:41,231 DEBUG HandlerThread:2117937 [handler.py:handle_request():158] handle_request: shutdown 2025-01-06 21:05:41,231 INFO HandlerThread:2117937 [handler.py:finish():882] shutting down handler 2025-01-06 21:05:42,167 INFO WriterThread:2117937 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/3d3i07nt/wandb/run-20250106_210053-3d3i07nt/run-3d3i07nt.wandb 2025-01-06 21:05:42,230 INFO SenderThread:2117937 [sender.py:finish():1615] shutting down sender 2025-01-06 21:05:42,230 INFO SenderThread:2117937 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 21:05:42,230 INFO SenderThread:2117937 [file_pusher.py:join():175] waiting for file pusher