2025-01-07 16:58:49,655 INFO StreamThr :2192706 [internal.py:wandb_internal():85] W&B internal server running at pid: 2192706, started at: 2025-01-07 16:58:49.654614 2025-01-07 16:58:49,655 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status 2025-01-07 16:58:49,668 INFO WriterThread:2192706 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/run-1exa8l2r.wandb 2025-01-07 16:58:49,668 DEBUG SenderThread:2192706 [sender.py:send():379] send: header 2025-01-07 16:58:49,675 DEBUG SenderThread:2192706 [sender.py:send():379] send: run 2025-01-07 16:58:49,962 INFO SenderThread:2192706 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files 2025-01-07 16:58:49,962 INFO SenderThread:2192706 [sender.py:_start_run_threads():1188] run started: 1exa8l2r with start time 1736287129.655871 2025-01-07 16:58:49,978 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: check_version 2025-01-07 16:58:49,979 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: check_version 2025-01-07 16:58:50,065 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: run_start 2025-01-07 16:58:50,070 DEBUG HandlerThread:2192706 [system_info.py:__init__():26] System info init 2025-01-07 16:58:50,070 DEBUG HandlerThread:2192706 [system_info.py:__init__():41] System info init done 2025-01-07 16:58:50,070 INFO HandlerThread:2192706 [system_monitor.py:start():194] Starting system monitor 2025-01-07 16:58:50,070 INFO SystemMonitor:2192706 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-07 16:58:50,070 INFO HandlerThread:2192706 [system_monitor.py:probe():214] Collecting system info 2025-01-07 16:58:50,071 INFO SystemMonitor:2192706 [interfaces.py:start():188] Started cpu monitoring 2025-01-07 16:58:50,071 INFO SystemMonitor:2192706 [interfaces.py:start():188] Started disk monitoring 2025-01-07 16:58:50,072 INFO SystemMonitor:2192706 [interfaces.py:start():188] Started gpu monitoring 2025-01-07 16:58:50,072 INFO SystemMonitor:2192706 [interfaces.py:start():188] Started memory monitoring 2025-01-07 16:58:50,073 INFO SystemMonitor:2192706 [interfaces.py:start():188] Started network monitoring 2025-01-07 16:58:50,090 DEBUG HandlerThread:2192706 [system_info.py:probe():152] Probing system 2025-01-07 16:58:50,091 DEBUG HandlerThread:2192706 [system_info.py:_probe_git():137] Probing git 2025-01-07 16:58:50,096 DEBUG HandlerThread:2192706 [system_info.py:_probe_git():145] Probing git done 2025-01-07 16:58:50,096 DEBUG HandlerThread:2192706 [system_info.py:probe():200] Probing system done 2025-01-07 16:58:50,096 DEBUG HandlerThread:2192706 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T21:58:50.090144', 'startedAt': '2025-01-07T21:58:49.641407', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3934.3185000000003, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.953, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 723.8112297058105}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-07 16:58:50,096 INFO HandlerThread:2192706 [system_monitor.py:probe():224] Finished collecting system info 2025-01-07 16:58:50,096 INFO HandlerThread:2192706 [system_monitor.py:probe():227] Publishing system info 2025-01-07 16:58:50,096 DEBUG HandlerThread:2192706 [system_info.py:_save_code():44] Saving code 2025-01-07 16:58:50,102 DEBUG HandlerThread:2192706 [system_info.py:_save_code():67] Saving code done 2025-01-07 16:58:50,102 DEBUG HandlerThread:2192706 [system_info.py:_save_patches():84] Saving git patches 2025-01-07 16:58:50,196 DEBUG HandlerThread:2192706 [system_info.py:_save_patches():126] Saving git patches done 2025-01-07 16:58:50,196 INFO HandlerThread:2192706 [system_monitor.py:probe():229] Finished publishing system info 2025-01-07 16:58:50,199 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 16:58:50,199 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-07 16:58:50,200 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-07 16:58:50,200 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-07 16:58:50,446 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/fx1h7118-wandb-metadata.json 2025-01-07 16:58:50,454 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: python_packages 2025-01-07 16:58:50,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: python_packages 2025-01-07 16:58:50,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 16:58:50,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 16:58:50,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:50,457 INFO HandlerThread:2192706 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/tb" save: true } control { relay_id: "8899518ba0484a498e6f13061035b9dd" } _info { stream_id: "1exa8l2r" } 2025-01-07 16:58:50,457 DEBUG HandlerThread:2192706 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-07 16:58:50,472 INFO wandb-upload_1:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/m7e3sa56-code/FISH/eval_robot.py 2025-01-07 16:58:50,483 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/dthq6sum-diff.patch 2025-01-07 16:58:50,600 DEBUG SenderThread:2192706 [sender.py:send():379] send: telemetry 2025-01-07 16:58:50,600 DEBUG SenderThread:2192706 [sender.py:send():379] send: tbrecord 2025-01-07 16:58:50,600 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 16:58:50,600 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 with policy live 2025-01-07 16:58:50,780 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/xkk90gpj-events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 16:58:50,963 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/diff.patch 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-metadata.json 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/code/FISH/eval_robot.py 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/requirements.txt 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/code/FISH 2025-01-07 16:58:50,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/code 2025-01-07 16:58:51,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:52,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:52,963 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:58:53,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:54,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:55,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:58:55,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:56,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:57,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:57,825 DEBUG SenderThread:2192706 [sender.py:send():379] send: config 2025-01-07 16:58:58,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:58:58,964 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:58:59,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:00,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:00,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:00,965 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:01,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:02,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:02,966 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:03,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:04,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:04,966 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:05,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 16:59:05,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 16:59:05,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:05,611 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:06,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:06,967 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:07,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:08,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:09,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:10,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:11,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:11,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:12,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:13,459 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:14,459 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:14,969 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:15,459 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:16,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:16,459 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:16,970 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:17,461 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:18,461 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:18,970 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:19,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:20,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 16:59:20,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 16:59:20,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:21,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:21,592 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:21,971 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/config.yaml 2025-01-07 16:59:22,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:23,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:24,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:25,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:26,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:27,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:27,689 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:28,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:28,973 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:29,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:30,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:31,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:32,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:33,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:33,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:34,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:35,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 16:59:35,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:35,498 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 16:59:36,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:36,981 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:37,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:38,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:38,602 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:39,464 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:40,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:41,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:42,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:43,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:44,200 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:44,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:45,152 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 16:59:45,152 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 with policy now 2025-01-07 16:59:45,152 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 16:59:45,153 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 16:59:45,153 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 16:59:45,154 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 16:59:45,404 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/1byh3xhq-media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 2025-01-07 16:59:45,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:45,984 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 2025-01-07 16:59:45,984 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 16:59:45,984 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media 2025-01-07 16:59:45,984 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos 2025-01-07 16:59:45,985 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval 2025-01-07 16:59:46,260 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/oo76nf8c-media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 2025-01-07 16:59:46,465 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:46,984 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:47,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:48,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:49,409 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:49,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:50,073 DEBUG SystemMonitor:2192706 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-07 16:59:50,074 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 16:59:50,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 16:59:50,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 16:59:50,501 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:50,986 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:51,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:52,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:52,986 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 16:59:53,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:54,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:55,118 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 16:59:55,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:56,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:57,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:58,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 16:59:59,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:00,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:00,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:01,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:02,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:03,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:04,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:05,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:00:05,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:00:05,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:05,523 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:06,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:07,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:08,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:09,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:10,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:11,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:11,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:12,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:12,993 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:00:13,237 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/dsn5nkcp-events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:00:13,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:14,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:14,994 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:15,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:16,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:16,994 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:17,222 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:17,470 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:18,470 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:18,995 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:19,470 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:20,075 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:00:20,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:00:20,456 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:00:20,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:21,470 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:22,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:22,574 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:23,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:24,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:25,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:26,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:27,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:28,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:28,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:29,471 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:30,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:31,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:32,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:33,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:33,797 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:34,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:35,000 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:35,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:00:35,455 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:00:35,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:36,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:37,001 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:37,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:38,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:39,002 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:39,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:39,797 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:40,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:41,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:42,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:43,003 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:43,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:44,385 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:00:44,385 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 with policy now 2025-01-07 17:00:44,388 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:00:44,388 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:00:44,388 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:00:44,395 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:00:44,474 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:44,639 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/lz0rmtpf-media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 2025-01-07 17:00:45,004 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:00:45,004 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:45,004 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 2025-01-07 17:00:45,004 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval 2025-01-07 17:00:45,280 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/sg1y8jde-media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 2025-01-07 17:00:45,401 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:45,476 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:46,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:47,004 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:47,482 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:48,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:49,005 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:49,482 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:50,075 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:00:50,421 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:50,469 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:00:50,469 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:00:50,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:51,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:52,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:53,007 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:53,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:54,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:55,459 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:00:55,480 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:56,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:56,890 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:00:56,890 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:00:56,890 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:00:57,008 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:00:57,009 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:00:57,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:58,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:00:59,009 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:00:59,482 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:00,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:00,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:01,010 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:01,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:02,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:03,011 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:03,481 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:04,482 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:05,455 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:01:05,456 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:01:05,497 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:05,581 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:06,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:07,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:08,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:09,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:10,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:11,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:11,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:12,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:13,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:14,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:15,484 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:16,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:17,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:17,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:18,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:19,016 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:19,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:20,076 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:01:20,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:01:20,456 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:01:20,501 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:21,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:22,486 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:22,543 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:23,486 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:24,486 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:25,487 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:26,486 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:27,488 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:27,889 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:28,243 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:01:28,243 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 with policy now 2025-01-07 17:01:28,248 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:01:28,249 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:01:28,249 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:01:28,249 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:01:28,488 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:28,499 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/0g8ccs9p-media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 2025-01-07 17:01:29,019 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:01:29,019 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:29,019 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 2025-01-07 17:01:29,020 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval 2025-01-07 17:01:29,323 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/fh1frayt-media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 2025-01-07 17:01:29,491 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:30,489 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:31,489 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:32,492 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:33,020 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:33,493 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:33,625 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:34,489 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:35,457 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:01:35,457 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:01:35,501 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:36,494 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:37,022 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:37,496 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:38,500 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:39,495 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:39,534 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:40,499 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:41,499 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:42,496 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:43,499 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:44,500 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:45,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:45,500 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:46,499 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:47,501 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:47,747 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:01:47,747 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:01:47,747 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:01:48,025 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:01:48,026 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:01:48,285 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/j7murppu-events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:01:48,506 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:49,026 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:49,505 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:50,076 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:01:50,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:01:50,458 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:01:50,506 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:50,537 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:51,027 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:51,612 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:52,599 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:53,027 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:01:53,598 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:54,598 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:55,598 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:56,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:01:56,598 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:57,599 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:58,598 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:01:59,599 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:00,599 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:01,599 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:02,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:02,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:03,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:04,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:05,456 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:02:05,457 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:02:05,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:06,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:07,594 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:07,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:08,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:09,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:10,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:11,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:12,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:13,467 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:13,600 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:14,601 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:15,601 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:16,601 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:17,601 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:18,577 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:18,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:19,604 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:19,713 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:02:19,713 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 with policy now 2025-01-07 17:02:19,716 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:02:19,716 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:02:19,716 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:02:19,717 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:02:19,972 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/bmos0zwk-media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 2025-01-07 17:02:20,036 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:02:20,036 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 2025-01-07 17:02:20,037 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval 2025-01-07 17:02:20,077 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:02:20,298 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/r6f9hx37-media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 2025-01-07 17:02:20,461 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:02:20,461 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:02:20,609 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:21,037 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:21,609 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:22,613 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:23,607 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:24,285 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:24,610 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:25,038 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:25,612 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:26,611 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:27,039 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:27,613 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:28,612 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:29,612 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:30,066 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:30,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:31,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:32,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:32,800 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:02:32,801 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:02:32,802 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:02:33,041 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:02:33,041 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:02:33,041 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:33,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:34,621 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:35,041 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:35,349 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:35,466 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:02:35,466 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:02:35,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:36,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:37,042 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:37,615 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:38,615 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:39,043 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:02:39,615 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:40,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:40,615 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:41,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:42,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:43,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:44,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:45,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:45,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:46,616 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:47,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:48,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:49,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:50,077 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:02:50,458 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:02:50,458 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:02:50,593 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:50,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:51,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:52,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:53,617 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:54,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:55,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:56,475 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:02:56,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:57,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:58,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:02:59,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:00,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:01,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:02,473 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:02,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:03,618 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:04,619 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:05,468 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:03:05,468 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:03:05,628 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:05,879 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:03:05,879 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 with policy now 2025-01-07 17:03:05,880 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:03:05,880 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:03:05,880 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:05,881 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:06,062 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:06,062 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 2025-01-07 17:03:06,062 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval 2025-01-07 17:03:06,138 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/6ccuuatk-media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 2025-01-07 17:03:06,395 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/ii264243-media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 2025-01-07 17:03:06,632 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:07,062 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:07,637 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:07,917 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:08,636 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:09,636 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:10,636 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:11,063 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:11,639 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:12,638 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:13,064 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:13,341 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:13,637 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:14,643 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:15,648 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:16,645 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:17,638 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:18,485 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:18,655 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:19,653 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:20,082 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:03:20,463 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:03:20,463 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:03:20,658 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:21,650 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:22,655 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:23,635 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:23,661 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:24,661 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:25,653 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:26,660 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:27,656 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:28,658 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:29,495 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:29,661 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:30,657 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:31,661 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:32,674 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:33,669 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:33,873 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:03:33,874 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:33,874 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:34,071 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:03:34,071 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:34,284 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 17:03:34,284 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 17:03:34,322 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/qnep3tbh-events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:03:34,462 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:03:34,665 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:34,779 INFO wandb-upload_0:2192706 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdfbhba_e 2025-01-07 17:03:34,845 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 17:03:34,853 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table 2025-01-07 17:03:35,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table/eval 2025-01-07 17:03:35,243 INFO SenderThread:2192706 [sender.py:send_request_log_artifact():1525] logged artifact run-1exa8l2r-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTg2NDQ4Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY4MzE2Nw==', 'latestArtifact': None}} 2025-01-07 17:03:35,243 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:03:35,243 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json with policy now 2025-01-07 17:03:35,243 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:35,243 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:03:35,244 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,244 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,244 DEBUG SenderThread:2192706 [sender.py:send():379] send: telemetry 2025-01-07 17:03:35,244 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,248 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,248 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,249 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,249 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,249 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,249 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,250 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,250 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,250 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,251 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,251 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,251 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,251 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,252 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,252 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,252 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,252 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,252 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,253 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,253 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:35,253 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:35,253 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 17:03:35,442 INFO wandb-upload_1:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/rnwxrymp-media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-07 17:03:35,453 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/pw46t55s-media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json 2025-01-07 17:03:35,460 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 17:03:35,665 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:35,689 INFO wandb-upload_1:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/idgarda9-media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-07 17:03:35,795 INFO wandb-upload_0:2192706 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp_j60bioa 2025-01-07 17:03:36,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:36,181 INFO SenderThread:2192706 [sender.py:send_request_log_artifact():1525] logged artifact run-1exa8l2r-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTg2NDUxNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY4MzE2OQ==', 'latestArtifact': None}} 2025-01-07 17:03:36,182 DEBUG SenderThread:2192706 [sender.py:send():379] send: files 2025-01-07 17:03:36,182 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json with policy now 2025-01-07 17:03:36,182 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:03:36,183 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:36,183 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:36,183 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: stop_status 2025-01-07 17:03:36,427 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/u9ans44y-media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json 2025-01-07 17:03:36,537 DEBUG SenderThread:2192706 [sender.py:send():379] send: exit 2025-01-07 17:03:36,537 INFO SenderThread:2192706 [sender.py:send_exit():586] handling exit code: 0 2025-01-07 17:03:36,538 INFO SenderThread:2192706 [sender.py:send_exit():588] handling runtime: 286 2025-01-07 17:03:36,538 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:36,538 INFO SenderThread:2192706 [sender.py:send_exit():594] send defer 2025-01-07 17:03:36,538 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:36,538 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-07 17:03:36,538 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:36,538 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-07 17:03:36,539 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 1 2025-01-07 17:03:36,539 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:36,539 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-07 17:03:36,539 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:36,539 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-07 17:03:36,539 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 2 2025-01-07 17:03:36,539 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:36,539 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-07 17:03:36,539 INFO HandlerThread:2192706 [system_monitor.py:finish():203] Stopping system monitor 2025-01-07 17:03:36,539 DEBUG SystemMonitor:2192706 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-07 17:03:36,539 DEBUG SystemMonitor:2192706 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-07 17:03:36,540 INFO HandlerThread:2192706 [interfaces.py:finish():200] Joined cpu monitor 2025-01-07 17:03:36,540 INFO HandlerThread:2192706 [interfaces.py:finish():200] Joined disk monitor 2025-01-07 17:03:36,593 INFO HandlerThread:2192706 [interfaces.py:finish():200] Joined gpu monitor 2025-01-07 17:03:36,593 INFO HandlerThread:2192706 [interfaces.py:finish():200] Joined memory monitor 2025-01-07 17:03:36,593 INFO HandlerThread:2192706 [interfaces.py:finish():200] Joined network monitor 2025-01-07 17:03:36,594 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:36,594 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-07 17:03:36,594 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 3 2025-01-07 17:03:36,594 DEBUG SenderThread:2192706 [sender.py:send():379] send: stats 2025-01-07 17:03:36,595 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:36,595 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-07 17:03:36,595 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:36,595 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-07 17:03:36,595 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 4 2025-01-07 17:03:36,595 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:36,595 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-07 17:03:37,072 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:37,073 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:39,073 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:42,880 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:42,881 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:42,881 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: keepalive 2025-01-07 17:03:42,881 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:42,882 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-07 17:03:42,882 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 5 2025-01-07 17:03:42,882 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:42,882 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:42,883 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-07 17:03:42,883 DEBUG SenderThread:2192706 [sender.py:send():379] send: history 2025-01-07 17:03:42,883 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: summary_record 2025-01-07 17:03:42,885 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:42,885 DEBUG SenderThread:2192706 [sender.py:send():379] send: summary 2025-01-07 17:03:42,885 INFO SenderThread:2192706 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 17:03:42,885 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:42,885 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-07 17:03:42,885 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 6 2025-01-07 17:03:42,885 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:42,886 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-07 17:03:42,886 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:42,886 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-07 17:03:42,889 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 17:03:43,012 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 7 2025-01-07 17:03:43,013 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:43,013 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-07 17:03:43,013 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:43,013 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-07 17:03:43,075 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:43,075 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/config.yaml 2025-01-07 17:03:43,539 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:46,471 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 8 2025-01-07 17:03:46,471 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:46,472 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:46,472 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-07 17:03:46,472 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:46,472 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-07 17:03:46,472 INFO SenderThread:2192706 [job_builder.py:build():440] Attempting to build job artifact 2025-01-07 17:03:46,474 INFO SenderThread:2192706 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-07 17:03:46,476 INFO SenderThread:2192706 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-07 17:03:46,490 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 9 2025-01-07 17:03:46,490 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:46,490 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-07 17:03:46,490 DEBUG SenderThread:2192706 [sender.py:send():379] send: artifact 2025-01-07 17:03:46,539 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:46,995 INFO wandb-upload_1:2192706 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpn5jjzskl 2025-01-07 17:03:46,995 INFO wandb-upload_2:2192706 [upload_job.py:push():85] Skipped uploading /tmp/tmpfj24jvix/wandb-job.json 2025-01-07 17:03:47,077 INFO Thread-12 :2192706 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:47,175 INFO wandb-upload_0:2192706 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpr5j4j1ba 2025-01-07 17:03:47,710 INFO SenderThread:2192706 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTg2NDkwOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTgyOTA5NQ==', 'versionIndex': 433}}} 2025-01-07 17:03:47,710 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:47,710 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-07 17:03:47,710 INFO SenderThread:2192706 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-07 17:03:48,077 INFO SenderThread:2192706 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files 2025-01-07 17:03:48,077 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/config.yaml config.yaml 2025-01-07 17:03:48,078 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json wandb-summary.json 2025-01-07 17:03:48,078 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:03:48,078 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/requirements.txt requirements.txt 2025-01-07 17:03:48,081 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-metadata.json wandb-metadata.json 2025-01-07 17:03:48,081 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log output.log 2025-01-07 17:03:48,081 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/diff.patch diff.patch 2025-01-07 17:03:48,081 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-07 17:03:48,082 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 media/videos/eval/0_eval_0_133b41ccfd01da169e6f.mp4 2025-01-07 17:03:48,082 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 media/videos/eval/3_eval_3_dc74b0d17439e756067e.mp4 2025-01-07 17:03:48,082 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 media/videos/eval/1_eval_1_46ad08891ecc2d017fe5.mp4 2025-01-07 17:03:48,082 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 media/videos/eval/4_eval_4_e22e33fe5bb8bbb0d7c4.mp4 2025-01-07 17:03:48,082 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 media/videos/eval/2_eval_2_2b1402d0e72f1d58d0bd.mp4 2025-01-07 17:03:48,084 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json media/table/eval/success_rate_ci_5_85c54431b4eb7c8145a7.table.json 2025-01-07 17:03:48,084 INFO SenderThread:2192706 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json media/table/eval/total_success_rate_ci_6_a80255a63fbf0577db61.table.json 2025-01-07 17:03:48,084 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 10 2025-01-07 17:03:48,084 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:48,084 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:48,084 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-07 17:03:48,087 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:48,087 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-07 17:03:48,087 INFO SenderThread:2192706 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 17:03:48,268 INFO wandb-upload_1:2192706 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/config.yaml 2025-01-07 17:03:48,276 INFO wandb-upload_2:2192706 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/wandb-summary.json 2025-01-07 17:03:48,289 INFO wandb-upload_0:2192706 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/requirements.txt 2025-01-07 17:03:48,292 INFO wandb-upload_4:2192706 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/files/output.log 2025-01-07 17:03:48,342 INFO wandb-upload_3:2192706 [upload_job.py:push():130] Uploaded file /tmp/tmprohsv1cgwandb/yh7a92kv-events.out.tfevents.1736287130.leonmkim-ROG-Strix-G15CS-G15CS.2192541.0 2025-01-07 17:03:48,540 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:48,541 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:48,542 INFO Thread-11 (_thread_body):2192706 [sender.py:transition_state():614] send defer: 11 2025-01-07 17:03:48,542 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:48,542 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-07 17:03:48,543 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:48,543 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-07 17:03:48,543 INFO SenderThread:2192706 [file_pusher.py:join():175] waiting for file pusher 2025-01-07 17:03:48,544 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 12 2025-01-07 17:03:48,544 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:48,544 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-07 17:03:48,544 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:48,544 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-07 17:03:48,544 INFO SenderThread:2192706 [file_stream.py:finish():601] file stream finish called 2025-01-07 17:03:48,711 INFO SenderThread:2192706 [file_stream.py:finish():605] file stream finish is done 2025-01-07 17:03:48,711 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 13 2025-01-07 17:03:48,711 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:48,711 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-07 17:03:48,712 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:48,712 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-07 17:03:48,712 INFO SenderThread:2192706 [sender.py:transition_state():614] send defer: 14 2025-01-07 17:03:48,712 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: defer 2025-01-07 17:03:48,712 INFO HandlerThread:2192706 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-07 17:03:48,712 DEBUG SenderThread:2192706 [sender.py:send():379] send: final 2025-01-07 17:03:48,712 DEBUG SenderThread:2192706 [sender.py:send():379] send: footer 2025-01-07 17:03:48,712 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: defer 2025-01-07 17:03:48,712 INFO SenderThread:2192706 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-07 17:03:48,713 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:48,713 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: server_info 2025-01-07 17:03:48,713 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 17:03:48,713 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:48,713 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: server_info 2025-01-07 17:03:48,715 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: get_summary 2025-01-07 17:03:48,715 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-07 17:03:48,716 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 17:03:48,769 DEBUG SenderThread:2192706 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 17:03:48,769 INFO MainThread:2192706 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-07 17:03:48,770 INFO MainThread:2192706 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-07 17:03:48,770 INFO MainThread:2192706 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-07 17:03:48,770 DEBUG HandlerThread:2192706 [handler.py:handle_request():158] handle_request: shutdown 2025-01-07 17:03:48,770 INFO HandlerThread:2192706 [handler.py:finish():882] shutting down handler 2025-01-07 17:03:49,713 INFO WriterThread:2192706 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/1exa8l2r/wandb/run-20250107_165849-1exa8l2r/run-1exa8l2r.wandb 2025-01-07 17:03:49,769 INFO SenderThread:2192706 [sender.py:finish():1615] shutting down sender 2025-01-07 17:03:49,769 INFO SenderThread:2192706 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 17:03:49,769 INFO SenderThread:2192706 [file_pusher.py:join():175] waiting for file pusher