2025-01-04 22:52:09,336 INFO StreamThr :2042123 [internal.py:wandb_internal():85] W&B internal server running at pid: 2042123, started at: 2025-01-04 22:52:09.336087 2025-01-04 22:52:09,337 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status 2025-01-04 22:52:09,350 INFO WriterThread:2042123 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/run-xt2ne5el.wandb 2025-01-04 22:52:09,351 DEBUG SenderThread:2042123 [sender.py:send():379] send: header 2025-01-04 22:52:09,358 DEBUG SenderThread:2042123 [sender.py:send():379] send: run 2025-01-04 22:52:09,566 INFO SenderThread:2042123 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files 2025-01-04 22:52:09,566 INFO SenderThread:2042123 [sender.py:_start_run_threads():1188] run started: xt2ne5el with start time 1736049129.336554 2025-01-04 22:52:09,581 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: check_version 2025-01-04 22:52:09,581 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: check_version 2025-01-04 22:52:09,628 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: run_start 2025-01-04 22:52:09,633 DEBUG HandlerThread:2042123 [system_info.py:__init__():26] System info init 2025-01-04 22:52:09,633 DEBUG HandlerThread:2042123 [system_info.py:__init__():41] System info init done 2025-01-04 22:52:09,633 INFO HandlerThread:2042123 [system_monitor.py:start():194] Starting system monitor 2025-01-04 22:52:09,634 INFO SystemMonitor:2042123 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-04 22:52:09,634 INFO SystemMonitor:2042123 [interfaces.py:start():188] Started cpu monitoring 2025-01-04 22:52:09,634 INFO SystemMonitor:2042123 [interfaces.py:start():188] Started disk monitoring 2025-01-04 22:52:09,635 INFO HandlerThread:2042123 [system_monitor.py:probe():214] Collecting system info 2025-01-04 22:52:09,635 INFO SystemMonitor:2042123 [interfaces.py:start():188] Started gpu monitoring 2025-01-04 22:52:09,636 INFO SystemMonitor:2042123 [interfaces.py:start():188] Started memory monitoring 2025-01-04 22:52:09,636 INFO SystemMonitor:2042123 [interfaces.py:start():188] Started network monitoring 2025-01-04 22:52:09,647 DEBUG HandlerThread:2042123 [system_info.py:probe():152] Probing system 2025-01-04 22:52:09,649 DEBUG HandlerThread:2042123 [system_info.py:_probe_git():137] Probing git 2025-01-04 22:52:09,654 DEBUG HandlerThread:2042123 [system_info.py:_probe_git():145] Probing git done 2025-01-04 22:52:09,654 DEBUG HandlerThread:2042123 [system_info.py:probe():200] Probing system done 2025-01-04 22:52:09,654 DEBUG HandlerThread:2042123 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-05T03:52:09.647525', 'startedAt': '2025-01-05T03:52:09.324843', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '02cf07d901f3d8187a302cd51c9faa80321417ee'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.7805000000003, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4498.264, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.007, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 787.8546829223633}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-04 22:52:09,654 INFO HandlerThread:2042123 [system_monitor.py:probe():224] Finished collecting system info 2025-01-04 22:52:09,654 INFO HandlerThread:2042123 [system_monitor.py:probe():227] Publishing system info 2025-01-04 22:52:09,654 DEBUG HandlerThread:2042123 [system_info.py:_save_code():44] Saving code 2025-01-04 22:52:09,655 DEBUG HandlerThread:2042123 [system_info.py:_save_code():67] Saving code done 2025-01-04 22:52:09,655 DEBUG HandlerThread:2042123 [system_info.py:_save_patches():84] Saving git patches 2025-01-04 22:52:09,718 DEBUG HandlerThread:2042123 [system_info.py:_save_patches():126] Saving git patches done 2025-01-04 22:52:09,718 INFO HandlerThread:2042123 [system_monitor.py:probe():229] Finished publishing system info 2025-01-04 22:52:09,721 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:52:09,721 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-04 22:52:09,721 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-04 22:52:09,722 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-04 22:52:09,949 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/v8ig4z7x-wandb-metadata.json 2025-01-04 22:52:09,962 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: python_packages 2025-01-04 22:52:09,962 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: python_packages 2025-01-04 22:52:09,964 DEBUG SenderThread:2042123 [sender.py:send():379] send: telemetry 2025-01-04 22:52:10,002 INFO wandb-upload_2:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/dp777x6a-diff.patch 2025-01-04 22:52:10,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:10,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:10,005 INFO HandlerThread:2042123 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/tb" save: true } control { relay_id: "b91b4588e7fe4d55a0ffb1941cedd315" } _info { stream_id: "xt2ne5el" } 2025-01-04 22:52:10,005 DEBUG HandlerThread:2042123 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-04 22:52:10,006 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:10,017 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/hwo6bjcq-code/FISH/eval_robot.py 2025-01-04 22:52:10,158 DEBUG SenderThread:2042123 [sender.py:send():379] send: tbrecord 2025-01-04 22:52:10,158 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:52:10,158 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 with policy live 2025-01-04 22:52:10,335 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/38td6c0n-events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-metadata.json 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/code/FISH/eval_robot.py 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/diff.patch 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/requirements.txt 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:52:10,568 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/code 2025-01-04 22:52:10,569 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/code/FISH 2025-01-04 22:52:10,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:11,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:12,567 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:12,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:13,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:14,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:14,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:15,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:16,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:17,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:18,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:19,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:19,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:20,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:21,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:22,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:23,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:24,962 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:24,963 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:25,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:25,051 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:25,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:26,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:27,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:28,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:29,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:30,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:30,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:31,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:32,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:33,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:34,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:35,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:35,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:36,966 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:37,871 DEBUG SenderThread:2042123 [sender.py:send():379] send: config 2025-01-04 22:52:37,966 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:38,966 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:39,962 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:39,963 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:40,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:40,574 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:40,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:41,087 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:41,575 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/config.yaml 2025-01-04 22:52:41,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:42,575 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:42,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:43,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:44,576 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:44,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:45,968 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:46,349 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:46,576 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:52:46,968 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:47,968 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:48,968 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:49,968 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:50,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:51,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:51,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:52,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:53,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:54,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:52:54,963 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:52:55,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:55,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:56,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:57,057 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:52:57,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:58,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:52:59,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:00,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:01,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:02,881 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:02,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:03,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:04,581 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:04,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:05,970 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:06,582 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:06,971 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:07,971 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:08,582 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:08,881 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:08,971 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:09,637 DEBUG SystemMonitor:2042123 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-04 22:53:09,638 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:53:09,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:09,963 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:09,971 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:10,972 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:11,972 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:12,583 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:12,973 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:13,973 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:14,461 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:14,584 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:14,973 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:15,977 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:16,584 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:16,977 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:17,981 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:18,981 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:19,466 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:19,981 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:20,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:21,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:22,586 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:22,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:23,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:24,963 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:24,963 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:25,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:25,071 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:25,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:26,984 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:27,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:28,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:29,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:30,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:30,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:31,986 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:32,986 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:33,782 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:53:33,782 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 with policy now 2025-01-04 22:53:33,783 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:53:33,784 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:53:33,784 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:53:33,787 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:53:33,986 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:34,066 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/cm9c9j2x-media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 2025-01-04 22:53:34,589 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:34,589 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:53:34,590 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 2025-01-04 22:53:34,590 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media 2025-01-04 22:53:34,590 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval 2025-01-04 22:53:34,590 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos 2025-01-04 22:53:34,864 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/7rspm7xz-media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 2025-01-04 22:53:34,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:35,966 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:35,986 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:36,590 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:36,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:37,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:38,591 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:38,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:39,638 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:53:39,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:39,964 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:39,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:40,591 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:40,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:41,036 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:41,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:42,592 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:53:42,853 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/wygl29c4-events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:53:42,987 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:43,592 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:53:43,988 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:44,593 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:44,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:45,990 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:46,326 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:46,988 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:47,988 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:48,594 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:48,988 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:49,988 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:50,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:51,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:51,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:52,595 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:52,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:53,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:54,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:53:54,964 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:53:55,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:55,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:56,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:57,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:53:57,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:58,598 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:53:58,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:53:59,990 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:00,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:01,989 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:02,990 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:03,966 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:03,990 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:04,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:05,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:06,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:07,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:08,967 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:08,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:09,638 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:54:09,964 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:09,964 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:10,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:10,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:11,991 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:12,992 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:13,996 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:14,084 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:14,344 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:54:14,344 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 with policy now 2025-01-04 22:54:14,347 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:54:14,347 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:54:14,347 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:54:14,348 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:54:14,598 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/v9a0sdvp-media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 2025-01-04 22:54:14,603 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:54:14,603 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:14,603 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 2025-01-04 22:54:14,603 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval 2025-01-04 22:54:14,894 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/bifo173a-media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 2025-01-04 22:54:14,999 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:15,996 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:16,603 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:17,001 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:18,001 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:18,604 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:18,998 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:19,813 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:20,008 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:20,604 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:20,997 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:22,008 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:22,605 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:23,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:23,606 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:54:24,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:24,440 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:54:24,441 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:54:24,441 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:54:24,606 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:54:24,606 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:24,974 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:24,976 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:24,976 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:25,017 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:26,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:26,607 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:27,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:28,003 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:28,607 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:29,003 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:30,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:30,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:31,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:32,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:33,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:34,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:35,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:35,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:36,004 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:37,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:38,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:38,611 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:39,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:39,639 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:54:39,965 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:39,965 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:40,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:41,005 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:41,068 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:42,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:43,006 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:44,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:45,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:46,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:46,281 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:46,613 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:47,007 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:48,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:49,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:50,009 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:51,010 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:51,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:52,010 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:53,010 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:54,010 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:54,971 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:54:54,972 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:54:55,012 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:55,257 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:54:55,257 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 with policy now 2025-01-04 22:54:55,258 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:54:55,258 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:54:55,259 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:54:55,261 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:54:55,519 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/ox298yj8-media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 2025-01-04 22:54:55,616 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:54:55,616 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 2025-01-04 22:54:55,617 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval 2025-01-04 22:54:55,886 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/e86uaaj7-media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 2025-01-04 22:54:56,013 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:56,617 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:54:57,018 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:57,277 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:54:58,021 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:54:59,016 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:00,015 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:00,618 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:01,018 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:02,018 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:02,619 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:03,027 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:03,261 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:04,019 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:05,034 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:06,024 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:07,025 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:08,027 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:08,978 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:09,027 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:09,639 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:55:09,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:09,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:10,035 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:11,032 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:12,029 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:13,030 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:14,030 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:14,067 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:15,032 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:16,034 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:16,280 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:55:16,280 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:55:16,280 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:55:16,623 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:55:16,623 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:55:16,861 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/wq2r75n7-events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:55:17,041 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:18,039 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:18,624 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:19,036 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:19,345 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:20,043 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:21,036 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:22,036 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:22,625 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:23,036 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:24,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:24,631 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:24,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:24,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:25,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:26,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:27,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:28,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:29,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:29,977 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:30,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:31,037 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:32,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:33,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:34,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:34,977 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:35,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:36,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:37,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:38,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:39,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:39,640 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:55:39,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:39,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:40,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:40,065 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:41,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:42,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:43,038 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:44,039 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:45,039 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:45,978 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:46,039 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:47,039 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:48,045 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:48,428 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:55:48,428 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 with policy now 2025-01-04 22:55:48,430 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:55:48,431 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:55:48,431 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:55:48,431 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:55:48,641 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:55:48,641 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:48,641 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 2025-01-04 22:55:48,642 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval 2025-01-04 22:55:48,687 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/lhcbg358-media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 2025-01-04 22:55:48,945 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/wf7yoy1l-media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 2025-01-04 22:55:49,047 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:50,049 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:50,642 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:50,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:51,050 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:52,048 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:52,642 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:53,055 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:54,057 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:54,643 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:54,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:55:54,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:55:55,053 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:56,059 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:56,121 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:55:56,644 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:55:57,069 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:58,067 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:55:59,067 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:00,066 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:01,071 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:01,982 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:02,073 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:03,074 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:04,066 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:05,069 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:06,068 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:06,992 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:07,072 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:08,070 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:09,092 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:09,640 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:56:09,975 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:09,975 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:10,087 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:11,086 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:12,089 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:12,116 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:13,092 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:14,091 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:15,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:16,107 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:16,356 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:56:16,357 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:16,357 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:16,650 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:56:16,651 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:16,651 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:56:16,902 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/605inuye-events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:56:17,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:17,861 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:18,107 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:18,651 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:19,100 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:20,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:20,652 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:21,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:22,096 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:22,653 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:22,924 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:23,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:24,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:24,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:24,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:25,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:26,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:27,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:28,055 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:28,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:29,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:30,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:31,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:32,097 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:33,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:33,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:34,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:35,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:36,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:37,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:38,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:38,985 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:39,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:39,641 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:56:39,969 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:39,969 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:40,098 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:41,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:42,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:43,099 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:44,100 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:44,128 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:45,101 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:46,100 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:47,101 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:48,101 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:49,103 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:49,440 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:49,703 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:56:49,703 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 with policy now 2025-01-04 22:56:49,704 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:56:49,704 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:56:49,704 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:56:49,705 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:56:49,963 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/q0wi346c-media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 2025-01-04 22:56:50,102 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:50,662 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:56:50,662 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:50,663 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 2025-01-04 22:56:50,663 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval 2025-01-04 22:56:50,956 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/do38pui6-media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 2025-01-04 22:56:51,104 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:52,107 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:53,117 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:54,109 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:54,664 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:54,972 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: stop_status 2025-01-04 22:56:54,972 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: stop_status 2025-01-04 22:56:55,057 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:56:55,128 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:56,127 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:57,119 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:58,127 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:56:58,665 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:56:59,119 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:00,121 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:00,326 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-04 22:57:00,326 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: log_artifact 2025-01-04 22:57:00,577 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:00,666 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:57:00,666 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:57:00,666 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json 2025-01-04 22:57:00,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media 2025-01-04 22:57:00,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval 2025-01-04 22:57:00,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table 2025-01-04 22:57:00,832 INFO wandb-upload_0:2042123 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdjmvylrk 2025-01-04 22:57:00,843 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/j7ylrfmb-media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json 2025-01-04 22:57:00,984 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-04 22:57:00,992 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: partial_history 2025-01-04 22:57:01,118 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:01,207 INFO SenderThread:2042123 [sender.py:send_request_log_artifact():1525] logged artifact run-xt2ne5el-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTIxMDc3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg2NjAwNQ==', 'latestArtifact': None}} 2025-01-04 22:57:01,207 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:57:01,207 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json with policy now 2025-01-04 22:57:01,208 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:57:01,208 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,208 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:01,209 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,209 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:57:01,209 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,209 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,210 DEBUG SenderThread:2042123 [sender.py:send():379] send: telemetry 2025-01-04 22:57:01,210 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,210 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,210 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,210 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,211 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,212 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,213 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,213 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,213 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,214 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,214 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,214 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,214 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,215 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,215 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,215 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,215 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,216 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,216 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,216 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,216 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,217 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:01,217 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: log_artifact 2025-01-04 22:57:01,446 INFO wandb-upload_2:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/wgf7axul-media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json 2025-01-04 22:57:01,598 INFO wandb-upload_1:2042123 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp49l7jzj8 2025-01-04 22:57:01,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:57:01,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:57:01,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json 2025-01-04 22:57:01,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval 2025-01-04 22:57:01,970 INFO SenderThread:2042123 [sender.py:send_request_log_artifact():1525] logged artifact run-xt2ne5el-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTIxMDc4OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg2NjAwNg==', 'latestArtifact': None}} 2025-01-04 22:57:01,970 DEBUG SenderThread:2042123 [sender.py:send():379] send: files 2025-01-04 22:57:01,971 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json with policy now 2025-01-04 22:57:01,971 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:57:01,971 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:01,971 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:02,118 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:02,142 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/uown77oo-media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json 2025-01-04 22:57:02,386 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/itemdwsk-media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json 2025-01-04 22:57:02,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:57:02,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:57:03,118 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:03,273 DEBUG SenderThread:2042123 [sender.py:send():379] send: exit 2025-01-04 22:57:03,273 INFO SenderThread:2042123 [sender.py:send_exit():586] handling exit code: 0 2025-01-04 22:57:03,274 INFO SenderThread:2042123 [sender.py:send_exit():588] handling runtime: 293 2025-01-04 22:57:03,274 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:03,274 INFO SenderThread:2042123 [sender.py:send_exit():594] send defer 2025-01-04 22:57:03,274 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:03,274 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-04 22:57:03,274 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:03,274 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-04 22:57:03,274 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 1 2025-01-04 22:57:03,275 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:03,275 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-04 22:57:03,275 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:03,275 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-04 22:57:03,275 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 2 2025-01-04 22:57:03,275 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:03,275 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-04 22:57:03,275 INFO HandlerThread:2042123 [system_monitor.py:finish():203] Stopping system monitor 2025-01-04 22:57:03,275 INFO HandlerThread:2042123 [interfaces.py:finish():200] Joined cpu monitor 2025-01-04 22:57:03,276 INFO HandlerThread:2042123 [interfaces.py:finish():200] Joined disk monitor 2025-01-04 22:57:03,276 DEBUG SystemMonitor:2042123 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-04 22:57:03,276 DEBUG SystemMonitor:2042123 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-04 22:57:03,327 INFO HandlerThread:2042123 [interfaces.py:finish():200] Joined gpu monitor 2025-01-04 22:57:03,328 INFO HandlerThread:2042123 [interfaces.py:finish():200] Joined memory monitor 2025-01-04 22:57:03,328 INFO HandlerThread:2042123 [interfaces.py:finish():200] Joined network monitor 2025-01-04 22:57:03,328 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:03,328 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-04 22:57:03,328 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 3 2025-01-04 22:57:03,328 DEBUG SenderThread:2042123 [sender.py:send():379] send: stats 2025-01-04 22:57:03,329 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:03,329 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-04 22:57:03,329 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:03,329 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-04 22:57:03,329 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 4 2025-01-04 22:57:03,329 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:03,329 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-04 22:57:03,667 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:57:04,668 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:57:09,420 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-04 22:57:09,420 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:09,420 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: keepalive 2025-01-04 22:57:09,421 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:09,421 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-04 22:57:09,421 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 5 2025-01-04 22:57:09,421 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: poll_exit 2025-01-04 22:57:09,422 DEBUG SenderThread:2042123 [sender.py:send():379] send: history 2025-01-04 22:57:09,422 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: summary_record 2025-01-04 22:57:09,422 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:09,422 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-04 22:57:09,423 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:09,423 DEBUG SenderThread:2042123 [sender.py:send():379] send: summary 2025-01-04 22:57:09,426 INFO SenderThread:2042123 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-04 22:57:09,426 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:09,427 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-04 22:57:09,427 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 6 2025-01-04 22:57:09,427 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:09,427 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-04 22:57:09,427 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:09,427 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-04 22:57:09,431 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: status_report 2025-01-04 22:57:09,529 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 7 2025-01-04 22:57:09,529 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:09,529 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-04 22:57:09,530 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:09,530 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-04 22:57:09,670 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:57:09,670 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/config.yaml 2025-01-04 22:57:10,274 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-04 22:57:10,472 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 8 2025-01-04 22:57:10,472 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: poll_exit 2025-01-04 22:57:10,472 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:10,472 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-04 22:57:10,473 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:10,473 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-04 22:57:10,473 INFO SenderThread:2042123 [job_builder.py:build():440] Attempting to build job artifact 2025-01-04 22:57:10,475 INFO SenderThread:2042123 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-04 22:57:10,477 INFO SenderThread:2042123 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-04 22:57:10,489 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 9 2025-01-04 22:57:10,490 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:10,490 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-04 22:57:10,490 DEBUG SenderThread:2042123 [sender.py:send():379] send: artifact 2025-01-04 22:57:10,655 INFO SenderThread:2042123 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxOTEzMDY5OQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxOTE5MjI0Ng==', 'versionIndex': 420}}} 2025-01-04 22:57:10,655 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:10,656 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-04 22:57:10,656 INFO SenderThread:2042123 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-04 22:57:10,670 INFO Thread-12 :2042123 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/config.yaml config.yaml 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json wandb-summary.json 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/requirements.txt requirements.txt 2025-01-04 22:57:10,671 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-metadata.json wandb-metadata.json 2025-01-04 22:57:10,672 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log output.log 2025-01-04 22:57:10,672 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/diff.patch diff.patch 2025-01-04 22:57:10,675 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 media/videos/eval/1_eval_1_537b6b4a557eb4c5962a.mp4 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 media/videos/eval/0_eval_0_fa08c14e432d2c4b05ac.mp4 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 media/videos/eval/4_eval_4_bf759c4daff03b0779d3.mp4 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 media/videos/eval/2_eval_2_6556ec2efb8093719953.mp4 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 media/videos/eval/3_eval_3_55177e1b719e0936946e.mp4 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json media/table/eval/success_rate_ci_5_e6508f0181aa84573747.table.json 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json media/table/eval/total_success_rate_ci_6_542b00ca2276b525cfd3.table.json 2025-01-04 22:57:10,680 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 10 2025-01-04 22:57:10,681 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:10,681 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-04 22:57:10,681 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:10,681 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-04 22:57:10,681 INFO SenderThread:2042123 [file_pusher.py:finish():169] shutting down file pusher 2025-01-04 22:57:10,858 INFO wandb-upload_0:2042123 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/config.yaml 2025-01-04 22:57:10,893 INFO wandb-upload_2:2042123 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/wandb-summary.json 2025-01-04 22:57:10,896 INFO wandb-upload_3:2042123 [upload_job.py:push():130] Uploaded file /tmp/tmps46638pxwandb/yx64pkx2-events.out.tfevents.1736049129.leonmkim-ROG-Strix-G15CS-G15CS.2041941.0 2025-01-04 22:57:10,902 INFO wandb-upload_1:2042123 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/requirements.txt 2025-01-04 22:57:10,913 INFO wandb-upload_4:2042123 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/files/output.log 2025-01-04 22:57:11,114 INFO Thread-11 (_thread_body):2042123 [sender.py:transition_state():614] send defer: 11 2025-01-04 22:57:11,114 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:11,114 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-04 22:57:11,114 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:11,114 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-04 22:57:11,114 INFO SenderThread:2042123 [file_pusher.py:join():175] waiting for file pusher 2025-01-04 22:57:11,117 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 12 2025-01-04 22:57:11,117 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:11,117 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-04 22:57:11,117 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:11,117 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-04 22:57:11,117 INFO SenderThread:2042123 [file_stream.py:finish():601] file stream finish called 2025-01-04 22:57:11,275 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-04 22:57:11,342 INFO SenderThread:2042123 [file_stream.py:finish():605] file stream finish is done 2025-01-04 22:57:11,342 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 13 2025-01-04 22:57:11,342 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: poll_exit 2025-01-04 22:57:11,342 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:11,342 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-04 22:57:11,343 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:11,343 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-04 22:57:11,343 INFO SenderThread:2042123 [sender.py:transition_state():614] send defer: 14 2025-01-04 22:57:11,343 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: defer 2025-01-04 22:57:11,343 INFO HandlerThread:2042123 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-04 22:57:11,343 DEBUG SenderThread:2042123 [sender.py:send():379] send: final 2025-01-04 22:57:11,343 DEBUG SenderThread:2042123 [sender.py:send():379] send: footer 2025-01-04 22:57:11,343 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: defer 2025-01-04 22:57:11,343 INFO SenderThread:2042123 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-04 22:57:11,344 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-04 22:57:11,344 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: server_info 2025-01-04 22:57:11,344 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: poll_exit 2025-01-04 22:57:11,344 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: server_info 2025-01-04 22:57:11,346 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-04 22:57:11,346 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: get_summary 2025-01-04 22:57:11,346 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-04 22:57:11,347 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-04 22:57:11,398 DEBUG SenderThread:2042123 [sender.py:send_request():406] send_request: poll_exit 2025-01-04 22:57:11,398 INFO MainThread:2042123 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-04 22:57:11,399 INFO MainThread:2042123 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-04 22:57:11,399 INFO MainThread:2042123 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-04 22:57:11,399 DEBUG HandlerThread:2042123 [handler.py:handle_request():158] handle_request: shutdown 2025-01-04 22:57:11,399 INFO HandlerThread:2042123 [handler.py:finish():882] shutting down handler 2025-01-04 22:57:12,346 INFO WriterThread:2042123 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/xt2ne5el/wandb/run-20250104_225209-xt2ne5el/run-xt2ne5el.wandb 2025-01-04 22:57:12,398 INFO SenderThread:2042123 [sender.py:finish():1615] shutting down sender 2025-01-04 22:57:12,398 INFO SenderThread:2042123 [file_pusher.py:finish():169] shutting down file pusher 2025-01-04 22:57:12,398 INFO SenderThread:2042123 [file_pusher.py:join():175] waiting for file pusher