2024-12-17 21:16:53,171 INFO StreamThr :75002 [internal.py:wandb_internal():85] W&B internal server running at pid: 75002, started at: 2024-12-17 21:16:53.170951 2024-12-17 21:16:53,172 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status 2024-12-17 21:16:53,183 INFO WriterThread:75002 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/run-14r6tvph.wandb 2024-12-17 21:16:53,183 DEBUG SenderThread:75002 [sender.py:send():379] send: header 2024-12-17 21:16:53,191 DEBUG SenderThread:75002 [sender.py:send():379] send: run 2024-12-17 21:16:53,401 INFO SenderThread:75002 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files 2024-12-17 21:16:53,401 INFO SenderThread:75002 [sender.py:_start_run_threads():1188] run started: 14r6tvph with start time 1734488213.172477 2024-12-17 21:16:53,414 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: check_version 2024-12-17 21:16:53,414 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: check_version 2024-12-17 21:16:53,486 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: run_start 2024-12-17 21:16:53,491 DEBUG HandlerThread:75002 [system_info.py:__init__():26] System info init 2024-12-17 21:16:53,491 DEBUG HandlerThread:75002 [system_info.py:__init__():41] System info init done 2024-12-17 21:16:53,491 INFO HandlerThread:75002 [system_monitor.py:start():194] Starting system monitor 2024-12-17 21:16:53,491 INFO SystemMonitor:75002 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-17 21:16:53,491 INFO HandlerThread:75002 [system_monitor.py:probe():214] Collecting system info 2024-12-17 21:16:53,492 INFO SystemMonitor:75002 [interfaces.py:start():188] Started cpu monitoring 2024-12-17 21:16:53,492 INFO SystemMonitor:75002 [interfaces.py:start():188] Started disk monitoring 2024-12-17 21:16:53,493 INFO SystemMonitor:75002 [interfaces.py:start():188] Started gpu monitoring 2024-12-17 21:16:53,493 INFO SystemMonitor:75002 [interfaces.py:start():188] Started memory monitoring 2024-12-17 21:16:53,494 INFO SystemMonitor:75002 [interfaces.py:start():188] Started network monitoring 2024-12-17 21:16:53,503 DEBUG HandlerThread:75002 [system_info.py:probe():152] Probing system 2024-12-17 21:16:53,505 DEBUG HandlerThread:75002 [system_info.py:_probe_git():137] Probing git 2024-12-17 21:16:53,510 DEBUG HandlerThread:75002 [system_info.py:_probe_git():145] Probing git done 2024-12-17 21:16:53,510 DEBUG HandlerThread:75002 [system_info.py:probe():200] Probing system done 2024-12-17 21:16:53,511 DEBUG HandlerThread:75002 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T02:16:53.503865', 'startedAt': '2024-12-18T02:16:53.161156', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3747.0996250000003, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4495.328, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.364, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 386.0898132324219}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2024-12-17 21:16:53,511 INFO HandlerThread:75002 [system_monitor.py:probe():224] Finished collecting system info 2024-12-17 21:16:53,511 INFO HandlerThread:75002 [system_monitor.py:probe():227] Publishing system info 2024-12-17 21:16:53,511 DEBUG HandlerThread:75002 [system_info.py:_save_code():44] Saving code 2024-12-17 21:16:53,512 DEBUG HandlerThread:75002 [system_info.py:_save_code():67] Saving code done 2024-12-17 21:16:53,512 DEBUG HandlerThread:75002 [system_info.py:_save_patches():84] Saving git patches 2024-12-17 21:16:53,585 DEBUG HandlerThread:75002 [system_info.py:_save_patches():126] Saving git patches done 2024-12-17 21:16:53,586 INFO HandlerThread:75002 [system_monitor.py:probe():229] Finished publishing system info 2024-12-17 21:16:53,589 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:16:53,589 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-17 21:16:53,589 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2024-12-17 21:16:53,589 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file diff.patch with policy now 2024-12-17 21:16:53,817 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: python_packages 2024-12-17 21:16:53,817 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: python_packages 2024-12-17 21:16:53,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:16:53,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:53,818 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:16:53,820 INFO HandlerThread:75002 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/tb" save: true } control { relay_id: "eed105154fd946b4aa5434f527304c55" } _info { stream_id: "14r6tvph" } 2024-12-17 21:16:53,820 DEBUG HandlerThread:75002 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2024-12-17 21:16:53,835 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/0eiaj0fa-diff.patch 2024-12-17 21:16:53,850 INFO wandb-upload_0:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/j4lpzn7m-wandb-metadata.json 2024-12-17 21:16:53,865 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/puflo4o7-code/FISH/eval_robot.py 2024-12-17 21:16:53,936 DEBUG SenderThread:75002 [sender.py:send():379] send: telemetry 2024-12-17 21:16:53,936 DEBUG SenderThread:75002 [sender.py:send():379] send: tbrecord 2024-12-17 21:16:53,936 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:16:53,937 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 with policy live 2024-12-17 21:16:54,105 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/8zd7zwie-events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:16:54,402 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/diff.patch 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-metadata.json 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/code/FISH/eval_robot.py 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/requirements.txt 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/code 2024-12-17 21:16:54,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/code/FISH 2024-12-17 21:16:54,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:55,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:56,402 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:16:56,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:57,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:58,660 DEBUG SenderThread:75002 [sender.py:send():379] send: config 2024-12-17 21:16:58,660 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:16:58,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:16:59,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:00,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:00,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:01,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:02,403 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:02,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:03,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:03,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:04,404 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:04,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:05,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:06,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:07,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:08,405 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:08,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:08,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:17:08,820 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:17:08,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:09,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:10,406 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:10,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:11,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:12,406 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:12,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:13,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:13,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:14,407 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:14,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:15,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:16,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:17,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:18,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:19,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:19,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:20,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:21,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:22,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:23,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:17:23,818 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:17:23,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:24,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:24,959 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:25,410 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/config.yaml 2024-12-17 21:17:25,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:26,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:27,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:28,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:29,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:30,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:30,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:31,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:32,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:33,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:34,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:35,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:36,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:36,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:37,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:38,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:17:38,818 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:17:38,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:39,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:40,802 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:17:40,802 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 with policy now 2024-12-17 21:17:40,803 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 21:17:40,803 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:17:40,804 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:17:40,805 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:17:40,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:41,058 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/htlcjn40-media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 2024-12-17 21:17:41,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:17:41,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 2024-12-17 21:17:41,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval 2024-12-17 21:17:41,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos 2024-12-17 21:17:41,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media 2024-12-17 21:17:41,664 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/7rdm6uyr-media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 2024-12-17 21:17:41,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:42,414 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:42,810 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:42,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:43,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:44,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:45,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:46,415 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:46,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:47,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:48,165 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:48,416 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:48,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:49,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:50,417 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:17:50,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:51,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:52,418 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:52,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:53,232 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:53,494 DEBUG SystemMonitor:75002 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-17 21:17:53,495 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:17:53,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:17:53,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:17:53,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:54,418 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:17:54,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:55,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:56,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:57,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:58,820 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:17:58,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:17:59,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:00,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:01,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:02,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:03,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:03,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:04,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:05,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:06,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:07,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:08,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:18:08,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:18:08,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:08,909 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:09,827 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:10,423 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:10,835 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:10,902 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:18:10,903 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 with policy now 2024-12-17 21:18:10,906 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 21:18:10,907 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:18:10,907 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:18:10,907 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:18:11,136 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/c3bv692d-media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 2024-12-17 21:18:11,424 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:18:11,424 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 2024-12-17 21:18:11,424 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval 2024-12-17 21:18:11,658 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/nduniaqo-media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 2024-12-17 21:18:11,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:12,424 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:12,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:13,424 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:13,840 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:14,127 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:14,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:15,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:16,425 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:16,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:17,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:18,426 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:18,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:19,821 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:19,838 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:20,838 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:21,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:22,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:23,501 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:18:23,818 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:18:23,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:18:23,865 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:24,838 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:24,907 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:25,838 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:26,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:27,838 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:28,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:29,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:30,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:30,840 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:31,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:32,840 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:33,840 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:34,840 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:35,841 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:36,822 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:36,845 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:37,843 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:38,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:18:38,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:18:38,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:39,843 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:40,843 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:41,849 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:41,895 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:42,843 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:43,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:44,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:45,248 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:18:45,248 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:18:45,249 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:18:45,435 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:18:45,435 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:18:45,660 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/5dz2mi6h-events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:18:45,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:46,435 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:46,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:47,649 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:47,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:48,436 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:48,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:49,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:50,437 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:18:50,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:51,844 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:52,823 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:52,845 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:53,496 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:18:53,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:18:53,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:18:53,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:54,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:55,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:56,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:57,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:57,946 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:18:58,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:18:59,846 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:00,440 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:00,863 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:01,859 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:02,189 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:19:02,189 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 with policy now 2024-12-17 21:19:02,191 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 21:19:02,192 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:19:02,192 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:19:02,192 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:19:02,427 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/2lqs5gyr-media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 2024-12-17 21:19:02,440 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:02,441 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:19:02,441 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 2024-12-17 21:19:02,441 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval 2024-12-17 21:19:02,689 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/d359f6i1-media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 2024-12-17 21:19:02,855 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:03,208 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:03,441 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:03,860 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:04,441 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:04,860 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:05,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:06,862 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:07,860 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:08,442 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:08,462 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:08,819 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:19:08,819 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:19:08,859 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:09,861 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:10,443 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:10,863 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:11,863 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:12,863 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:13,828 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:13,866 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:14,863 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:15,866 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:16,867 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:17,880 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:18,829 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:18,879 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:19,883 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:20,047 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:19:20,048 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:19:20,057 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:19:20,446 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:19:20,446 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:19:20,880 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:21,881 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:22,447 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:22,883 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:23,496 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:19:23,826 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:19:23,826 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:19:23,883 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:23,943 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:24,448 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:24,886 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:25,886 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:26,448 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:26,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:27,888 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:28,449 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:28,886 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:29,833 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:29,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:30,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:31,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:32,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:33,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:34,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:35,830 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:35,887 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:36,452 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:36,888 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:37,888 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:38,452 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:38,547 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:19:38,547 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 with policy now 2024-12-17 21:19:38,549 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 21:19:38,550 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:19:38,550 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:19:38,550 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:19:38,769 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/gbinh9n5-media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 2024-12-17 21:19:38,825 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:19:38,825 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:19:38,900 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:39,453 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:19:39,453 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 2024-12-17 21:19:39,453 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval 2024-12-17 21:19:39,831 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/3yzpie16-media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 2024-12-17 21:19:39,899 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:40,453 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:40,893 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:40,942 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:41,896 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:42,901 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:43,906 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:44,454 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:44,901 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:45,905 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:45,953 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:46,455 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:46,905 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:47,917 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:48,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:49,905 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:50,913 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:51,839 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:51,907 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:52,911 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:53,497 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:19:53,828 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:19:53,828 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:19:53,913 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:54,921 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:55,087 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:19:55,087 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:19:55,088 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:19:55,458 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:19:55,458 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:19:55,701 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/4lo3xwe5-events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:19:55,922 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:56,458 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:56,922 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:57,081 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:19:57,928 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:58,459 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:19:58,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:19:59,930 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:00,459 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:00,941 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:01,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:02,266 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:02,460 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:02,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:03,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:04,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:05,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:06,923 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:07,833 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:07,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:08,824 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:20:08,824 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:20:08,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:09,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:10,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:11,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:12,463 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:12,924 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:13,390 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:13,937 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:14,464 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:14,619 DEBUG SenderThread:75002 [sender.py:send():379] send: files 2024-12-17 21:20:14,619 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 with policy now 2024-12-17 21:20:14,622 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 21:20:14,622 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:20:14,622 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:20:14,622 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:20:14,849 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/1gbzb2e6-media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 2024-12-17 21:20:14,934 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:15,464 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:20:15,465 INFO Thread-12 :75002 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 2024-12-17 21:20:15,465 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval 2024-12-17 21:20:15,729 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/194n5l0n-media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 2024-12-17 21:20:15,933 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:16,465 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:16,937 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:17,935 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:18,836 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:18,963 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:19,936 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:20,466 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:20,939 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:21,950 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:22,467 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:22,942 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:23,497 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:20:23,837 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 21:20:23,837 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: stop_status 2024-12-17 21:20:23,950 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:23,954 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:24,938 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:25,949 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:26,977 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:27,948 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:28,955 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:29,850 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:29,961 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:30,952 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:31,955 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:32,953 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:33,956 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:34,957 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:35,121 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:35,131 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:20:35,131 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:20:35,132 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:20:35,165 DEBUG SenderThread:75002 [sender.py:send():379] send: exit 2024-12-17 21:20:35,165 INFO SenderThread:75002 [sender.py:send_exit():586] handling exit code: 0 2024-12-17 21:20:35,166 INFO SenderThread:75002 [sender.py:send_exit():588] handling runtime: 221 2024-12-17 21:20:35,166 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:20:35,166 INFO SenderThread:75002 [sender.py:send_exit():594] send defer 2024-12-17 21:20:35,166 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:35,166 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-17 21:20:35,166 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:35,166 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-17 21:20:35,166 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 1 2024-12-17 21:20:35,166 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:35,166 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-17 21:20:35,167 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:35,167 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-17 21:20:35,167 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 2 2024-12-17 21:20:35,167 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:35,167 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-17 21:20:35,167 INFO HandlerThread:75002 [system_monitor.py:finish():203] Stopping system monitor 2024-12-17 21:20:35,167 DEBUG SystemMonitor:75002 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-17 21:20:35,167 DEBUG SystemMonitor:75002 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-17 21:20:35,167 INFO HandlerThread:75002 [interfaces.py:finish():200] Joined cpu monitor 2024-12-17 21:20:35,168 INFO HandlerThread:75002 [interfaces.py:finish():200] Joined disk monitor 2024-12-17 21:20:35,217 INFO HandlerThread:75002 [interfaces.py:finish():200] Joined gpu monitor 2024-12-17 21:20:35,218 INFO HandlerThread:75002 [interfaces.py:finish():200] Joined memory monitor 2024-12-17 21:20:35,218 INFO HandlerThread:75002 [interfaces.py:finish():200] Joined network monitor 2024-12-17 21:20:35,218 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:35,218 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-17 21:20:35,218 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 3 2024-12-17 21:20:35,218 DEBUG SenderThread:75002 [sender.py:send():379] send: stats 2024-12-17 21:20:35,219 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:35,219 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-17 21:20:35,219 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:35,219 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-17 21:20:35,219 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 4 2024-12-17 21:20:35,219 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:35,219 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-17 21:20:35,471 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:20:35,471 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:20:36,471 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:41,220 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 21:20:41,220 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:41,220 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:41,221 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: keepalive 2024-12-17 21:20:41,221 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-17 21:20:41,221 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 5 2024-12-17 21:20:41,221 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 21:20:41,221 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:41,221 DEBUG SenderThread:75002 [sender.py:send():379] send: history 2024-12-17 21:20:41,221 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-17 21:20:41,222 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: summary_record 2024-12-17 21:20:41,222 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:20:41,222 DEBUG SenderThread:75002 [sender.py:send():379] send: summary 2024-12-17 21:20:41,226 INFO SenderThread:75002 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 21:20:41,226 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:41,226 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-17 21:20:41,226 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 6 2024-12-17 21:20:41,226 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:41,226 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-17 21:20:41,226 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:41,227 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-17 21:20:41,227 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 7 2024-12-17 21:20:41,227 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 21:20:41,227 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:41,227 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-17 21:20:41,227 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:41,227 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-17 21:20:41,473 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:20:42,166 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 21:20:42,831 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 8 2024-12-17 21:20:42,831 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 21:20:42,831 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:42,832 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-17 21:20:42,832 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:42,832 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-17 21:20:42,832 INFO SenderThread:75002 [job_builder.py:build():440] Attempting to build job artifact 2024-12-17 21:20:42,834 INFO SenderThread:75002 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-17 21:20:42,849 INFO SenderThread:75002 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-17 21:20:42,859 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 9 2024-12-17 21:20:42,859 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:42,859 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-17 21:20:42,860 DEBUG SenderThread:75002 [sender.py:send():379] send: artifact 2024-12-17 21:20:43,167 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 21:20:43,275 INFO wandb-upload_0:75002 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp58wukwcy 2024-12-17 21:20:43,276 INFO wandb-upload_2:75002 [upload_job.py:push():85] Skipped uploading /tmp/tmpyn9jqz0t/wandb-job.json 2024-12-17 21:20:43,474 INFO Thread-12 :75002 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:43,479 INFO wandb-upload_1:75002 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpvalkza0y 2024-12-17 21:20:44,190 INFO SenderThread:75002 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzE5ODcyNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzE0NTc2Mg==', 'versionIndex': 377}}} 2024-12-17 21:20:44,190 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,190 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-17 21:20:44,190 INFO SenderThread:75002 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-17 21:20:44,475 INFO SenderThread:75002 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files 2024-12-17 21:20:44,475 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/config.yaml config.yaml 2024-12-17 21:20:44,475 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json wandb-summary.json 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/requirements.txt requirements.txt 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-metadata.json wandb-metadata.json 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log output.log 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/diff.patch diff.patch 2024-12-17 21:20:44,477 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2024-12-17 21:20:44,483 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 media/videos/eval/3_eval_3_d96484f372adc32d22ea.mp4 2024-12-17 21:20:44,483 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 media/videos/eval/0_eval_0_607b63808c40640027f8.mp4 2024-12-17 21:20:44,484 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 media/videos/eval/1_eval_1_528c8c53973fc263e4c2.mp4 2024-12-17 21:20:44,484 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 media/videos/eval/2_eval_2_2f040d8fedc4c652052d.mp4 2024-12-17 21:20:44,484 INFO SenderThread:75002 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 media/videos/eval/4_eval_4_85f8c5555fdc1daec6f7.mp4 2024-12-17 21:20:44,484 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 10 2024-12-17 21:20:44,484 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 21:20:44,484 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:44,484 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-17 21:20:44,484 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,484 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-17 21:20:44,484 INFO SenderThread:75002 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 21:20:44,647 INFO wandb-upload_3:75002 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/wandb-summary.json 2024-12-17 21:20:44,653 INFO wandb-upload_0:75002 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/config.yaml 2024-12-17 21:20:44,691 INFO wandb-upload_4:75002 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/output.log 2024-12-17 21:20:44,695 INFO wandb-upload_2:75002 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/files/requirements.txt 2024-12-17 21:20:44,698 INFO wandb-upload_1:75002 [upload_job.py:push():130] Uploaded file /tmp/tmp_bvp7plcwandb/eajpd6gl-events.out.tfevents.1734488213.leonmkim-ROG-Strix-G15CS-G15CS.74816.0 2024-12-17 21:20:44,898 INFO Thread-11 (_thread_body):75002 [sender.py:transition_state():614] send defer: 11 2024-12-17 21:20:44,899 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:44,899 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-17 21:20:44,899 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,899 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-17 21:20:44,899 INFO SenderThread:75002 [file_pusher.py:join():175] waiting for file pusher 2024-12-17 21:20:44,900 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 12 2024-12-17 21:20:44,900 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:44,900 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-17 21:20:44,900 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,900 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-17 21:20:44,900 INFO SenderThread:75002 [file_stream.py:finish():601] file stream finish called 2024-12-17 21:20:44,962 INFO SenderThread:75002 [file_stream.py:finish():605] file stream finish is done 2024-12-17 21:20:44,962 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 13 2024-12-17 21:20:44,962 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:44,962 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-17 21:20:44,962 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,962 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-17 21:20:44,962 INFO SenderThread:75002 [sender.py:transition_state():614] send defer: 14 2024-12-17 21:20:44,962 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: defer 2024-12-17 21:20:44,963 DEBUG SenderThread:75002 [sender.py:send():379] send: final 2024-12-17 21:20:44,963 INFO HandlerThread:75002 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-17 21:20:44,963 DEBUG SenderThread:75002 [sender.py:send():379] send: footer 2024-12-17 21:20:44,963 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: defer 2024-12-17 21:20:44,963 INFO SenderThread:75002 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-17 21:20:44,963 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 21:20:44,963 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 21:20:44,964 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: server_info 2024-12-17 21:20:44,964 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: server_info 2024-12-17 21:20:44,965 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 21:20:44,965 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: get_summary 2024-12-17 21:20:44,966 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-17 21:20:44,966 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 21:20:45,040 DEBUG SenderThread:75002 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 21:20:45,040 INFO MainThread:75002 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2024-12-17 21:20:45,041 INFO MainThread:75002 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2024-12-17 21:20:45,041 INFO MainThread:75002 [wandb_run.py:_footer_sync_info():3975] logging synced files 2024-12-17 21:20:45,041 DEBUG HandlerThread:75002 [handler.py:handle_request():158] handle_request: shutdown 2024-12-17 21:20:45,041 INFO HandlerThread:75002 [handler.py:finish():882] shutting down handler 2024-12-17 21:20:45,966 INFO WriterThread:75002 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/508_0/211647/wandb/run-20241217_211653-14r6tvph/run-14r6tvph.wandb 2024-12-17 21:20:46,040 INFO SenderThread:75002 [sender.py:finish():1615] shutting down sender 2024-12-17 21:20:46,041 INFO SenderThread:75002 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 21:20:46,041 INFO SenderThread:75002 [file_pusher.py:join():175] waiting for file pusher