2024-12-17 20:50:37,561 INFO StreamThr :69178 [internal.py:wandb_internal():85] W&B internal server running at pid: 69178, started at: 2024-12-17 20:50:37.560794 2024-12-17 20:50:37,561 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status 2024-12-17 20:50:37,573 INFO WriterThread:69178 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/run-v5toezqj.wandb 2024-12-17 20:50:37,573 DEBUG SenderThread:69178 [sender.py:send():379] send: header 2024-12-17 20:50:37,580 DEBUG SenderThread:69178 [sender.py:send():379] send: run 2024-12-17 20:50:37,815 INFO SenderThread:69178 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files 2024-12-17 20:50:37,815 INFO SenderThread:69178 [sender.py:_start_run_threads():1188] run started: v5toezqj with start time 1734486637.561221 2024-12-17 20:50:37,831 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: check_version 2024-12-17 20:50:37,831 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: check_version 2024-12-17 20:50:37,909 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: run_start 2024-12-17 20:50:37,914 DEBUG HandlerThread:69178 [system_info.py:__init__():26] System info init 2024-12-17 20:50:37,914 DEBUG HandlerThread:69178 [system_info.py:__init__():41] System info init done 2024-12-17 20:50:37,914 INFO HandlerThread:69178 [system_monitor.py:start():194] Starting system monitor 2024-12-17 20:50:37,914 INFO SystemMonitor:69178 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-17 20:50:37,914 INFO HandlerThread:69178 [system_monitor.py:probe():214] Collecting system info 2024-12-17 20:50:37,915 INFO SystemMonitor:69178 [interfaces.py:start():188] Started cpu monitoring 2024-12-17 20:50:37,915 INFO SystemMonitor:69178 [interfaces.py:start():188] Started disk monitoring 2024-12-17 20:50:37,916 INFO SystemMonitor:69178 [interfaces.py:start():188] Started gpu monitoring 2024-12-17 20:50:37,916 INFO SystemMonitor:69178 [interfaces.py:start():188] Started memory monitoring 2024-12-17 20:50:37,917 INFO SystemMonitor:69178 [interfaces.py:start():188] Started network monitoring 2024-12-17 20:50:37,929 DEBUG HandlerThread:69178 [system_info.py:probe():152] Probing system 2024-12-17 20:50:37,930 DEBUG HandlerThread:69178 [system_info.py:_probe_git():137] Probing git 2024-12-17 20:50:37,935 DEBUG HandlerThread:69178 [system_info.py:_probe_git():145] Probing git done 2024-12-17 20:50:37,935 DEBUG HandlerThread:69178 [system_info.py:probe():200] Probing system done 2024-12-17 20:50:37,935 DEBUG HandlerThread:69178 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T01:50:37.929248', 'startedAt': '2024-12-18T01:50:37.553164', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.51775, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.895, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.968, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 360.83281326293945}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2024-12-17 20:50:37,935 INFO HandlerThread:69178 [system_monitor.py:probe():224] Finished collecting system info 2024-12-17 20:50:37,935 INFO HandlerThread:69178 [system_monitor.py:probe():227] Publishing system info 2024-12-17 20:50:37,936 DEBUG HandlerThread:69178 [system_info.py:_save_code():44] Saving code 2024-12-17 20:50:37,937 DEBUG HandlerThread:69178 [system_info.py:_save_code():67] Saving code done 2024-12-17 20:50:37,937 DEBUG HandlerThread:69178 [system_info.py:_save_patches():84] Saving git patches 2024-12-17 20:50:38,004 DEBUG HandlerThread:69178 [system_info.py:_save_patches():126] Saving git patches done 2024-12-17 20:50:38,005 INFO HandlerThread:69178 [system_monitor.py:probe():229] Finished publishing system info 2024-12-17 20:50:38,008 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:50:38,008 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-17 20:50:38,008 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2024-12-17 20:50:38,008 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file diff.patch with policy now 2024-12-17 20:50:38,246 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: python_packages 2024-12-17 20:50:38,246 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: python_packages 2024-12-17 20:50:38,246 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:50:38,247 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:50:38,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:38,293 INFO HandlerThread:69178 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/tb" save: true } control { relay_id: "f5f4f6fa4a0e46c6a826af22cb762514" } _info { stream_id: "v5toezqj" } 2024-12-17 20:50:38,293 DEBUG HandlerThread:69178 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2024-12-17 20:50:38,367 DEBUG SenderThread:69178 [sender.py:send():379] send: telemetry 2024-12-17 20:50:38,368 DEBUG SenderThread:69178 [sender.py:send():379] send: tbrecord 2024-12-17 20:50:38,368 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:50:38,368 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 with policy live 2024-12-17 20:50:38,386 INFO wandb-upload_2:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/1louuc8s-diff.patch 2024-12-17 20:50:38,387 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/mtmc0peb-wandb-metadata.json 2024-12-17 20:50:38,387 INFO wandb-upload_1:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/i4p6vweu-code/FISH/eval_robot.py 2024-12-17 20:50:38,570 INFO wandb-upload_3:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/fq8htgz7-events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:50:38,817 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/code/FISH/eval_robot.py 2024-12-17 20:50:38,817 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-metadata.json 2024-12-17 20:50:38,817 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/requirements.txt 2024-12-17 20:50:38,818 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/diff.patch 2024-12-17 20:50:38,818 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:38,818 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:50:38,818 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/code 2024-12-17 20:50:38,818 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/code/FISH 2024-12-17 20:50:39,247 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:40,247 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:40,817 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:41,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:42,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:43,247 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:50:43,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:44,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:45,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:46,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:47,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:48,247 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:50:48,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:49,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:50,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:51,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:51,545 DEBUG SenderThread:69178 [sender.py:send():379] send: config 2024-12-17 20:50:52,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:52,820 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:53,246 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:50:53,247 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:50:53,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:53,369 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:50:54,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:54,820 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:55,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:56,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:56,821 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:57,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:58,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:50:58,822 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:50:58,978 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:50:59,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:00,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:00,822 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:01,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:02,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:03,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:04,153 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:04,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:05,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:06,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:06,824 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:07,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:08,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:51:08,249 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:51:08,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:08,824 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:09,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:09,400 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:09,825 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/config.yaml 2024-12-17 20:51:10,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:10,825 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:11,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:12,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:13,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:14,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:15,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:15,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:16,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:17,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:18,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:19,253 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:20,248 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:20,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:21,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:22,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:23,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:51:23,249 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:51:23,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:24,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:25,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:25,350 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:26,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:27,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:28,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:29,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:30,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:31,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:31,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:32,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:33,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:34,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:35,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:35,589 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:51:35,589 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 with policy now 2024-12-17 20:51:35,589 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 20:51:35,590 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:51:35,590 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:51:35,591 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:51:35,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 2024-12-17 20:51:35,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:51:35,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media 2024-12-17 20:51:35,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos 2024-12-17 20:51:35,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval 2024-12-17 20:51:35,858 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/aemlia3l-media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 2024-12-17 20:51:36,164 INFO wandb-upload_1:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/370593m2-media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 2024-12-17 20:51:36,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:36,592 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:36,832 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:37,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:37,917 DEBUG SystemMonitor:69178 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-17 20:51:37,917 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:51:38,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:51:38,252 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:51:38,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:39,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:40,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:40,833 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:41,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:41,804 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:42,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:42,834 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:43,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:43,834 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:51:44,122 INFO wandb-upload_2:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/2lsrgr80-events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:51:44,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:44,835 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:45,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:46,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:46,835 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:47,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:47,398 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:48,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:48,836 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:51:49,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:50,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:51,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:52,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:53,249 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:53,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:51:53,250 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:51:53,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:54,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:55,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:56,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:57,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:58,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:51:58,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:51:59,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:00,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:01,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:02,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:03,250 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:03,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:04,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:05,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:06,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:07,259 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:07,918 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:52:08,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:08,257 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:52:08,257 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:52:08,301 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:09,259 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:10,259 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:11,259 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:12,259 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:13,260 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:14,130 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:14,260 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:15,262 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:15,570 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:52:15,570 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 with policy now 2024-12-17 20:52:15,576 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 20:52:15,577 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:52:15,577 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:52:15,578 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:52:15,832 INFO wandb-upload_3:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/q6lbjbni-media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 2024-12-17 20:52:15,844 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:52:15,845 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 2024-12-17 20:52:15,845 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval 2024-12-17 20:52:16,129 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/inolq7gm-media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 2024-12-17 20:52:16,263 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:16,845 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:17,262 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:17,845 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:18,263 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:19,137 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:19,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:20,263 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:20,846 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:21,270 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:22,263 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:22,847 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:23,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:52:23,251 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:52:23,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:23,681 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:52:23,681 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:52:23,703 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:52:23,847 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:52:23,847 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:52:24,264 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:24,294 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:24,848 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:25,269 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:26,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:26,848 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:27,266 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:28,266 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:28,849 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:29,266 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:30,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:30,266 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:31,266 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:32,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:33,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:34,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:35,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:36,254 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:36,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:37,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:37,919 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:52:38,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:52:38,251 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:52:38,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:39,267 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:40,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:41,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:41,337 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:42,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:43,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:44,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:45,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:46,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:47,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:47,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:48,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:49,268 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:50,269 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:51,269 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:52,269 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:53,251 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:52:53,251 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:52:53,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:53,398 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:54,274 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:54,857 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:55,182 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:52:55,182 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 with policy now 2024-12-17 20:52:55,182 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 20:52:55,183 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:52:55,183 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:52:55,183 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:52:55,274 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:55,446 INFO wandb-upload_1:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/rklbn3xb-media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 2024-12-17 20:52:55,858 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:52:55,859 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 2024-12-17 20:52:55,859 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval 2024-12-17 20:52:56,129 INFO wandb-upload_2:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/hbutwpli-media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 2024-12-17 20:52:56,281 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:56,859 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:57,289 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:57,859 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:52:58,276 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:52:58,711 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:52:59,281 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:00,295 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:00,860 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:01,292 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:02,286 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:02,861 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:03,291 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:03,861 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:53:04,105 INFO wandb-upload_3:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/dvaq4j8i-events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:53:04,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:04,510 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:53:04,510 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:53:04,511 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:04,511 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:53:04,862 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:53:04,862 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:05,293 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:06,295 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:06,862 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:07,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:07,919 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:53:08,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:53:08,252 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:53:08,294 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:08,863 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:09,296 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:09,937 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:10,296 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:10,864 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:11,296 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:12,296 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:13,296 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:14,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:14,865 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:15,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:15,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:16,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:17,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:18,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:19,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:20,255 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:20,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:21,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:22,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:23,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:53:23,252 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:53:23,297 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:24,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:25,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:25,414 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:26,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:27,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:28,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:29,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:30,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:31,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:31,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:32,298 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:33,299 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:34,299 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:35,301 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:35,372 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:53:35,372 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 with policy now 2024-12-17 20:53:35,413 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 20:53:35,414 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:53:35,414 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:53:35,415 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:53:35,629 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/us394crz-media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 2024-12-17 20:53:35,872 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:53:35,872 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 2024-12-17 20:53:35,872 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval 2024-12-17 20:53:36,136 INFO wandb-upload_1:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/c8thr94g-media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 2024-12-17 20:53:36,309 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:36,415 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:36,872 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:37,306 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:37,919 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:53:38,269 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:53:38,269 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:53:38,352 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:39,316 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:40,318 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:40,874 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:41,330 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:41,718 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:42,331 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:42,874 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:43,328 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:43,875 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:53:44,327 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:44,557 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:53:44,557 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:53:44,558 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:53:44,875 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:53:44,875 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:45,319 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:46,319 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:46,876 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:47,327 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:47,452 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:48,320 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:48,876 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:49,341 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:50,321 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:50,877 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:51,321 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:52,321 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:53,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:53:53,252 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:53:53,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:53,406 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:54,321 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:55,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:56,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:57,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:58,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:53:58,880 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:53:59,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:53:59,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:00,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:01,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:02,322 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:03,323 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:04,323 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:05,258 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:05,323 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:06,323 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:07,323 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:07,920 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:54:08,252 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:54:08,252 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:54:08,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:08,883 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:09,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:10,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:10,391 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:11,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:12,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:13,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:14,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:15,324 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:15,940 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:16,325 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:16,886 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:17,032 DEBUG SenderThread:69178 [sender.py:send():379] send: files 2024-12-17 20:54:17,032 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 with policy now 2024-12-17 20:54:17,035 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 20:54:17,035 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:54:17,035 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:54:17,036 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:54:17,297 INFO wandb-upload_2:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/07e8vmtr-media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 2024-12-17 20:54:17,331 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:17,886 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:54:17,886 INFO Thread-12 :69178 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 2024-12-17 20:54:17,887 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval 2024-12-17 20:54:18,176 INFO wandb-upload_3:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/c23rhxtl-media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 2024-12-17 20:54:18,336 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:18,887 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:19,335 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:20,330 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:21,335 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:21,617 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:22,334 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:22,888 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:23,256 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 20:54:23,257 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: stop_status 2024-12-17 20:54:23,344 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:24,348 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:24,889 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:25,317 DEBUG SenderThread:69178 [sender.py:send():379] send: exit 2024-12-17 20:54:25,317 INFO SenderThread:69178 [sender.py:send_exit():586] handling exit code: 0 2024-12-17 20:54:25,317 INFO SenderThread:69178 [sender.py:send_exit():588] handling runtime: 227 2024-12-17 20:54:25,318 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:54:25,318 INFO SenderThread:69178 [sender.py:send_exit():594] send defer 2024-12-17 20:54:25,318 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:25,318 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-17 20:54:25,318 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:25,318 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-17 20:54:25,318 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 1 2024-12-17 20:54:25,318 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:25,318 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-17 20:54:25,319 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:25,319 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-17 20:54:25,319 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 2 2024-12-17 20:54:25,319 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:25,319 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-17 20:54:25,319 INFO HandlerThread:69178 [system_monitor.py:finish():203] Stopping system monitor 2024-12-17 20:54:25,319 INFO HandlerThread:69178 [interfaces.py:finish():200] Joined cpu monitor 2024-12-17 20:54:25,319 INFO HandlerThread:69178 [interfaces.py:finish():200] Joined disk monitor 2024-12-17 20:54:25,320 DEBUG SystemMonitor:69178 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-17 20:54:25,320 DEBUG SystemMonitor:69178 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-17 20:54:25,366 INFO HandlerThread:69178 [interfaces.py:finish():200] Joined gpu monitor 2024-12-17 20:54:25,366 INFO HandlerThread:69178 [interfaces.py:finish():200] Joined memory monitor 2024-12-17 20:54:25,366 INFO HandlerThread:69178 [interfaces.py:finish():200] Joined network monitor 2024-12-17 20:54:25,367 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:25,367 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-17 20:54:25,367 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 3 2024-12-17 20:54:25,367 DEBUG SenderThread:69178 [sender.py:send():379] send: stats 2024-12-17 20:54:25,367 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:25,367 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-17 20:54:25,367 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:25,367 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-17 20:54:25,367 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 4 2024-12-17 20:54:25,367 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:25,367 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-17 20:54:25,889 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:54:25,889 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:54:26,148 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /tmp/tmphf0ydbrfwandb/f5qq7aq2-events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:54:26,889 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:31,369 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:31,370 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:31,370 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:31,370 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: keepalive 2024-12-17 20:54:31,370 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-17 20:54:31,370 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 5 2024-12-17 20:54:31,370 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:54:31,370 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:31,370 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:54:31,371 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-17 20:54:31,372 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:54:31,372 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:31,372 DEBUG SenderThread:69178 [sender.py:send():379] send: history 2024-12-17 20:54:31,372 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: summary_record 2024-12-17 20:54:31,375 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:54:31,376 DEBUG SenderThread:69178 [sender.py:send():379] send: summary 2024-12-17 20:54:31,376 INFO SenderThread:69178 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 20:54:31,376 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:31,376 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-17 20:54:31,376 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 6 2024-12-17 20:54:31,376 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:31,376 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-17 20:54:31,376 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:31,376 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-17 20:54:31,376 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 7 2024-12-17 20:54:31,376 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 20:54:31,377 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:31,377 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-17 20:54:31,377 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:31,377 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-17 20:54:31,891 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:54:32,319 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:33,581 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 8 2024-12-17 20:54:33,581 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:33,581 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:33,581 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-17 20:54:33,582 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:33,582 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-17 20:54:33,582 INFO SenderThread:69178 [job_builder.py:build():440] Attempting to build job artifact 2024-12-17 20:54:33,584 INFO SenderThread:69178 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-17 20:54:33,598 INFO SenderThread:69178 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-17 20:54:33,612 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 9 2024-12-17 20:54:33,612 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:33,612 DEBUG SenderThread:69178 [sender.py:send():379] send: artifact 2024-12-17 20:54:33,612 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-17 20:54:33,892 INFO Thread-12 :69178 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:34,275 INFO wandb-upload_3:69178 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp0pq2joja 2024-12-17 20:54:34,275 INFO wandb-upload_2:69178 [upload_job.py:push():85] Skipped uploading /tmp/tmpb66nr4i8/wandb-job.json 2024-12-17 20:54:34,320 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:34,372 INFO wandb-upload_1:69178 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp9m88tfvz 2024-12-17 20:54:34,948 INFO SenderThread:69178 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzA1OTA4Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4Mjk5MjQ0NQ==', 'versionIndex': 373}}} 2024-12-17 20:54:34,948 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:34,948 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-17 20:54:34,948 INFO SenderThread:69178 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-17 20:54:35,893 INFO SenderThread:69178 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files 2024-12-17 20:54:35,893 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/config.yaml config.yaml 2024-12-17 20:54:35,893 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json wandb-summary.json 2024-12-17 20:54:35,893 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 events.out.tfevents.1734486638.leonmkim-ROG-Strix-G15CS-G15CS.68994.0 2024-12-17 20:54:35,893 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/requirements.txt requirements.txt 2024-12-17 20:54:35,895 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-metadata.json wandb-metadata.json 2024-12-17 20:54:35,896 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log output.log 2024-12-17 20:54:35,897 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/diff.patch diff.patch 2024-12-17 20:54:35,899 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2024-12-17 20:54:35,899 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 media/videos/eval/1_eval_1_3dd750e4041b7d940696.mp4 2024-12-17 20:54:35,899 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 media/videos/eval/4_eval_4_cb4b291fcf155dd3f366.mp4 2024-12-17 20:54:35,899 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 media/videos/eval/3_eval_3_c5032837d4220bef66c0.mp4 2024-12-17 20:54:35,900 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 media/videos/eval/2_eval_2_41e781c681092c8d0dae.mp4 2024-12-17 20:54:35,900 INFO SenderThread:69178 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 media/videos/eval/0_eval_0_479c2ab590e51b63c1e7.mp4 2024-12-17 20:54:35,900 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 10 2024-12-17 20:54:35,900 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:35,900 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:35,900 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-17 20:54:35,900 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:35,900 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-17 20:54:35,900 INFO SenderThread:69178 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 20:54:36,085 INFO wandb-upload_2:69178 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/wandb-summary.json 2024-12-17 20:54:36,091 INFO wandb-upload_3:69178 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/requirements.txt 2024-12-17 20:54:36,093 INFO wandb-upload_0:69178 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/config.yaml 2024-12-17 20:54:36,158 INFO wandb-upload_1:69178 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/files/output.log 2024-12-17 20:54:36,320 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:36,321 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:36,358 INFO Thread-11 (_thread_body):69178 [sender.py:transition_state():614] send defer: 11 2024-12-17 20:54:36,358 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:36,359 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-17 20:54:36,359 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:36,359 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-17 20:54:36,359 INFO SenderThread:69178 [file_pusher.py:join():175] waiting for file pusher 2024-12-17 20:54:36,360 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 12 2024-12-17 20:54:36,360 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:36,360 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-17 20:54:36,360 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:36,360 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-17 20:54:36,360 INFO SenderThread:69178 [file_stream.py:finish():601] file stream finish called 2024-12-17 20:54:36,567 INFO SenderThread:69178 [file_stream.py:finish():605] file stream finish is done 2024-12-17 20:54:36,567 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 13 2024-12-17 20:54:36,567 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:36,567 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-17 20:54:36,567 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:36,567 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-17 20:54:36,567 INFO SenderThread:69178 [sender.py:transition_state():614] send defer: 14 2024-12-17 20:54:36,567 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: defer 2024-12-17 20:54:36,567 DEBUG SenderThread:69178 [sender.py:send():379] send: final 2024-12-17 20:54:36,568 INFO HandlerThread:69178 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-17 20:54:36,568 DEBUG SenderThread:69178 [sender.py:send():379] send: footer 2024-12-17 20:54:36,568 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: defer 2024-12-17 20:54:36,568 INFO SenderThread:69178 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-17 20:54:36,568 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:36,568 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:36,568 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: server_info 2024-12-17 20:54:36,569 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: server_info 2024-12-17 20:54:36,570 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 20:54:36,570 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: get_summary 2024-12-17 20:54:36,570 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-17 20:54:36,571 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 20:54:36,626 DEBUG SenderThread:69178 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 20:54:36,626 INFO MainThread:69178 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2024-12-17 20:54:36,626 INFO MainThread:69178 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2024-12-17 20:54:36,626 INFO MainThread:69178 [wandb_run.py:_footer_sync_info():3975] logging synced files 2024-12-17 20:54:36,627 DEBUG HandlerThread:69178 [handler.py:handle_request():158] handle_request: shutdown 2024-12-17 20:54:36,627 INFO HandlerThread:69178 [handler.py:finish():882] shutting down handler 2024-12-17 20:54:37,570 INFO WriterThread:69178 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/505_0/205031/wandb/run-20241217_205037-v5toezqj/run-v5toezqj.wandb 2024-12-17 20:54:37,626 INFO SenderThread:69178 [sender.py:finish():1615] shutting down sender 2024-12-17 20:54:37,626 INFO SenderThread:69178 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 20:54:37,626 INFO SenderThread:69178 [file_pusher.py:join():175] waiting for file pusher