2025-01-07 21:43:32,668 INFO StreamThr :2236235 [internal.py:wandb_internal():85] W&B internal server running at pid: 2236235, started at: 2025-01-07 21:43:32.667989 2025-01-07 21:43:32,669 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status 2025-01-07 21:43:32,680 INFO WriterThread:2236235 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/run-b3d2fbi3.wandb 2025-01-07 21:43:32,680 DEBUG SenderThread:2236235 [sender.py:send():379] send: header 2025-01-07 21:43:32,697 DEBUG SenderThread:2236235 [sender.py:send():379] send: run 2025-01-07 21:43:32,936 INFO SenderThread:2236235 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files 2025-01-07 21:43:32,936 INFO SenderThread:2236235 [sender.py:_start_run_threads():1188] run started: b3d2fbi3 with start time 1736304212.668334 2025-01-07 21:43:32,954 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: check_version 2025-01-07 21:43:32,954 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: check_version 2025-01-07 21:43:33,001 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: run_start 2025-01-07 21:43:33,006 DEBUG HandlerThread:2236235 [system_info.py:__init__():26] System info init 2025-01-07 21:43:33,006 DEBUG HandlerThread:2236235 [system_info.py:__init__():41] System info init done 2025-01-07 21:43:33,006 INFO HandlerThread:2236235 [system_monitor.py:start():194] Starting system monitor 2025-01-07 21:43:33,006 INFO SystemMonitor:2236235 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-07 21:43:33,006 INFO HandlerThread:2236235 [system_monitor.py:probe():214] Collecting system info 2025-01-07 21:43:33,007 INFO SystemMonitor:2236235 [interfaces.py:start():188] Started cpu monitoring 2025-01-07 21:43:33,007 INFO SystemMonitor:2236235 [interfaces.py:start():188] Started disk monitoring 2025-01-07 21:43:33,008 INFO SystemMonitor:2236235 [interfaces.py:start():188] Started gpu monitoring 2025-01-07 21:43:33,008 INFO SystemMonitor:2236235 [interfaces.py:start():188] Started memory monitoring 2025-01-07 21:43:33,009 INFO SystemMonitor:2236235 [interfaces.py:start():188] Started network monitoring 2025-01-07 21:43:33,020 DEBUG HandlerThread:2236235 [system_info.py:probe():152] Probing system 2025-01-07 21:43:33,021 DEBUG HandlerThread:2236235 [system_info.py:_probe_git():137] Probing git 2025-01-07 21:43:33,026 DEBUG HandlerThread:2236235 [system_info.py:_probe_git():145] Probing git done 2025-01-07 21:43:33,026 DEBUG HandlerThread:2236235 [system_info.py:probe():200] Probing system done 2025-01-07 21:43:33,026 DEBUG HandlerThread:2236235 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T02:43:33.020498', 'startedAt': '2025-01-08T02:43:32.660554', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.495875, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}, {'current': 4499.971, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 631.3639907836914}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-07 21:43:33,026 INFO HandlerThread:2236235 [system_monitor.py:probe():224] Finished collecting system info 2025-01-07 21:43:33,026 INFO HandlerThread:2236235 [system_monitor.py:probe():227] Publishing system info 2025-01-07 21:43:33,026 DEBUG HandlerThread:2236235 [system_info.py:_save_code():44] Saving code 2025-01-07 21:43:33,028 DEBUG HandlerThread:2236235 [system_info.py:_save_code():67] Saving code done 2025-01-07 21:43:33,028 DEBUG HandlerThread:2236235 [system_info.py:_save_patches():84] Saving git patches 2025-01-07 21:43:33,091 DEBUG HandlerThread:2236235 [system_info.py:_save_patches():126] Saving git patches done 2025-01-07 21:43:33,092 INFO HandlerThread:2236235 [system_monitor.py:probe():229] Finished publishing system info 2025-01-07 21:43:33,096 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:43:33,096 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-07 21:43:33,097 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-07 21:43:33,097 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-07 21:43:33,330 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: python_packages 2025-01-07 21:43:33,330 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: python_packages 2025-01-07 21:43:33,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:43:33,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:43:33,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:33,333 INFO HandlerThread:2236235 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/tb" save: true } control { relay_id: "a0fd80047fef4aa1a87eb5d9c6445f55" } _info { stream_id: "b3d2fbi3" } 2025-01-07 21:43:33,333 DEBUG HandlerThread:2236235 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-07 21:43:33,360 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/c4k5h5k6-wandb-metadata.json 2025-01-07 21:43:33,361 INFO wandb-upload_2:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/tp13inhx-diff.patch 2025-01-07 21:43:33,370 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/sl0pkjbn-code/FISH/eval_robot.py 2025-01-07 21:43:33,456 DEBUG SenderThread:2236235 [sender.py:send():379] send: telemetry 2025-01-07 21:43:33,456 DEBUG SenderThread:2236235 [sender.py:send():379] send: tbrecord 2025-01-07 21:43:33,457 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:43:33,457 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 with policy live 2025-01-07 21:43:33,635 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/ufifbwrf-events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/diff.patch 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/requirements.txt 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-metadata.json 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:43:33,938 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/code/FISH/eval_robot.py 2025-01-07 21:43:33,939 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/code 2025-01-07 21:43:33,939 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/code/FISH 2025-01-07 21:43:34,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:35,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:35,937 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:43:36,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:37,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:38,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:43:38,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:39,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:40,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:41,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:42,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:43,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:43:43,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:44,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:45,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:46,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:47,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:48,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:43:48,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:43:48,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:48,385 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:43:49,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:50,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:51,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:52,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:53,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:54,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:43:54,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:55,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:56,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:57,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:58,335 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:58,870 DEBUG SenderThread:2236235 [sender.py:send():379] send: config 2025-01-07 21:43:59,335 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:43:59,913 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:43:59,944 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:00,335 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:01,335 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:01,944 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:02,335 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:03,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:44:03,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:44:03,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:04,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:05,249 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:05,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:05,945 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:05,946 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/config.yaml 2025-01-07 21:44:06,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:07,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:07,946 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:08,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:09,361 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:10,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:11,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:11,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:11,947 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:12,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:13,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:13,948 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:14,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:15,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:16,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:16,385 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:17,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:18,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:44:18,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:44:18,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:19,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:20,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:21,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:21,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:22,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:23,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:24,339 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:25,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:26,339 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:27,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:27,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:28,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:29,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:30,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:31,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:32,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:32,953 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:33,009 DEBUG SystemMonitor:2236235 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-07 21:44:33,010 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:44:33,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:44:33,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:44:33,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:33,954 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:34,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:35,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:36,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:37,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:38,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:38,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:39,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:40,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:41,015 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:44:41,015 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 with policy now 2025-01-07 21:44:41,015 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:44:41,016 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:44:41,016 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:44:41,016 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:44:41,275 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/xo0lcz4d-media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 2025-01-07 21:44:41,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:41,956 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:41,956 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:44:41,956 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 2025-01-07 21:44:41,957 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media 2025-01-07 21:44:41,957 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos 2025-01-07 21:44:41,957 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval 2025-01-07 21:44:42,242 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/b32zjxno-media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 2025-01-07 21:44:42,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:43,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:43,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:44,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:45,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:45,958 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:46,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:47,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:48,331 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:44:48,331 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:44:48,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:48,462 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:49,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:49,959 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:44:50,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:51,342 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:52,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:53,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:54,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:54,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:55,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:56,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:57,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:58,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:44:59,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:44:59,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:00,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:01,343 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:02,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:03,010 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:45:03,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:45:03,332 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:45:03,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:04,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:04,456 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:05,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:06,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:07,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:08,344 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:09,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:10,337 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:10,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:11,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:12,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:13,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:14,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:15,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:16,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:16,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:17,345 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:18,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:45:18,332 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:45:18,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:19,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:20,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:21,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:21,484 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:22,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:23,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:24,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:25,346 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:26,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:27,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:27,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:28,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:29,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:30,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:31,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:32,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:32,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:33,011 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:45:33,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:45:33,332 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:45:33,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:34,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:35,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:36,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:37,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:37,429 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:38,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:39,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:40,353 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:40,975 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:45:41,236 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/a4tnxcgv-events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:45:41,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:41,976 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:45:42,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:43,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:43,348 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:43,977 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:45:44,349 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:45,349 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:45,977 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:45:46,349 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:47,349 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:48,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:45:48,332 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:45:48,377 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:48,402 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:49,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:50,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:51,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:52,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:53,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:54,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:45:54,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:55,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:56,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:57,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:58,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:45:59,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:00,340 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:00,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:01,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:02,351 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:03,015 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:46:03,332 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:46:03,332 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:46:03,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:04,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:05,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:05,418 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:06,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:07,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:08,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:09,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:10,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:11,072 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:11,356 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:11,985 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:12,363 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:12,484 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:46:12,484 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 with policy now 2025-01-07 21:46:12,487 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:46:12,487 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:46:12,487 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:46:12,488 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:46:12,730 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/gkwvirdk-media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 2025-01-07 21:46:12,986 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:46:12,986 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 2025-01-07 21:46:12,986 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval 2025-01-07 21:46:13,299 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/308ybxp2-media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 2025-01-07 21:46:13,358 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:13,986 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:14,362 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:14,986 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:15,361 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:16,362 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:16,953 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:17,360 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:17,987 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:18,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:46:18,338 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:46:18,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:19,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:19,988 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:20,364 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:21,364 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:22,366 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:22,738 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:23,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:24,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:25,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:26,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:26,833 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:46:26,833 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:46:26,834 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:46:26,990 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:46:26,991 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:46:27,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:27,991 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:28,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:28,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:29,365 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:29,992 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:30,368 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:31,366 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:31,992 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:46:32,366 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:33,012 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:46:33,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:46:33,333 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:46:33,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:33,392 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:34,367 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:35,368 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:36,368 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:37,368 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:38,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:39,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:39,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:40,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:41,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:42,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:43,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:44,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:44,369 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:45,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:46,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:47,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:48,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:46:48,333 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:46:48,377 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:49,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:49,446 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:50,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:51,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:52,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:53,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:54,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:55,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:46:55,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:56,370 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:57,371 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:58,380 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:58,998 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:46:58,998 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 with policy now 2025-01-07 21:46:59,001 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:46:59,001 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:46:59,002 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:46:59,002 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:46:59,003 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:46:59,003 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 2025-01-07 21:46:59,003 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval 2025-01-07 21:46:59,245 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/nfz5wxwi-media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 2025-01-07 21:46:59,371 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:46:59,509 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/osxrpxsr-media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 2025-01-07 21:47:00,003 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:00,372 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:01,004 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:01,027 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:01,373 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:02,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:03,012 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:47:03,338 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:47:03,338 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:47:03,376 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:04,005 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:04,379 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:05,375 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:06,265 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:06,376 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:07,387 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:08,006 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:08,384 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:09,378 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:10,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:11,354 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:11,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:12,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:12,664 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:47:12,664 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:47:12,665 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:47:13,008 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:47:13,008 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:47:13,259 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/ot33qyk0-events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:47:13,380 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:14,008 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:14,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:15,385 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:16,009 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:16,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:16,533 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:17,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:18,009 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:18,334 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:47:18,334 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:47:18,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:19,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:20,381 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:21,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:22,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:22,494 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:23,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:24,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:25,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:26,382 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:27,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:28,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:28,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:29,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:30,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:31,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:32,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:33,013 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:47:33,333 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:47:33,334 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:47:33,383 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:33,489 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:34,384 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:35,384 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:36,384 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:37,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:38,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:39,347 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:39,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:40,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:41,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:42,386 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:43,387 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:44,397 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:44,536 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:44,581 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:47:44,581 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 with policy now 2025-01-07 21:47:44,582 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:47:44,583 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:47:44,583 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:47:44,583 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:47:44,873 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/68rclkh2-media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 2025-01-07 21:47:45,019 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:47:45,019 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 2025-01-07 21:47:45,020 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval 2025-01-07 21:47:45,301 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/56syekz3-media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 2025-01-07 21:47:45,390 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:46,020 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:46,395 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:47,391 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:48,341 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:47:48,341 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:47:48,399 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:49,397 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:50,021 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:50,395 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:50,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:51,397 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:52,022 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:52,397 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:53,397 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:54,396 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:55,399 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:56,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:47:56,399 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:57,023 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:47:57,399 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:57,716 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:47:57,716 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:47:57,717 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:47:58,024 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:47:58,024 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:47:58,403 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:47:59,403 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:00,025 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:00,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:01,401 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:02,025 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:02,063 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:02,401 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:03,014 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:48:03,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:48:03,336 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:48:03,401 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:04,401 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:05,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:06,027 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:06,403 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:07,403 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:07,662 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:08,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:09,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:10,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:11,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:12,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:13,352 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:13,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:14,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:15,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:16,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:17,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:18,336 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:48:18,336 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:48:18,404 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:18,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:19,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:20,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:21,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:22,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:23,405 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:24,353 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:24,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:25,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:26,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:27,406 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:28,418 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:29,190 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:48:29,190 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 with policy now 2025-01-07 21:48:29,193 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:48:29,194 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:48:29,194 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:29,195 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:29,414 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:29,432 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/nk1lragn-media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 2025-01-07 21:48:30,035 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:48:30,035 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:30,035 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 2025-01-07 21:48:30,035 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval 2025-01-07 21:48:30,243 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:30,323 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/5dn17vtg-media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 2025-01-07 21:48:30,417 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:31,035 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:31,417 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:32,414 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:33,015 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:48:33,350 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:48:33,350 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:48:33,434 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:34,036 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:34,416 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:35,417 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:35,477 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:36,420 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:37,426 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:38,038 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:38,433 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:39,436 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:40,439 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:40,730 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:40,769 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:48:40,769 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:40,770 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:41,039 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:48:41,039 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:48:41,267 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 21:48:41,267 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 21:48:41,324 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/5ixqlopn-events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:48:41,411 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:48:41,428 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:41,704 INFO wandb-upload_0:2236235 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp_d48b567 2025-01-07 21:48:41,810 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-07 21:48:41,819 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: partial_history 2025-01-07 21:48:42,039 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:48:42,039 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:42,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json 2025-01-07 21:48:42,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 21:48:42,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media 2025-01-07 21:48:42,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table 2025-01-07 21:48:42,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table/eval 2025-01-07 21:48:42,098 INFO SenderThread:2236235 [sender.py:send_request_log_artifact():1525] logged artifact run-b3d2fbi3-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjUwMTM1OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg3MDAyMA==', 'latestArtifact': None}} 2025-01-07 21:48:42,098 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:48:42,098 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now 2025-01-07 21:48:42,098 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:48:42,099 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,099 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,099 DEBUG SenderThread:2236235 [sender.py:send():379] send: telemetry 2025-01-07 21:48:42,099 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,103 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,103 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,104 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,104 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,104 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,104 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,105 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,105 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,105 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,105 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,105 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,106 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,106 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,106 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,106 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,107 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,107 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,107 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,107 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,107 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,108 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:42,108 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: log_artifact 2025-01-07 21:48:42,279 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/wradrxg9-media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json 2025-01-07 21:48:42,310 INFO wandb-upload_2:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/0llywt9z-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 21:48:42,429 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:42,511 INFO wandb-upload_0:2236235 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpfxixy7rt 2025-01-07 21:48:42,676 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/b6sxtv2x-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 21:48:42,895 INFO SenderThread:2236235 [sender.py:send_request_log_artifact():1525] logged artifact run-b3d2fbi3-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjUwMTM2NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg3MDAyMg==', 'latestArtifact': None}} 2025-01-07 21:48:42,896 DEBUG SenderThread:2236235 [sender.py:send():379] send: files 2025-01-07 21:48:42,896 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json with policy now 2025-01-07 21:48:42,896 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:48:42,896 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:48:42,896 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:48:43,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:48:43,153 INFO wandb-upload_2:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/hu9y0g27-media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json 2025-01-07 21:48:43,429 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:44,040 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:48:44,429 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:45,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:46,358 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:46,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:47,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:48,339 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:48:48,339 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:48:48,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:49,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:50,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:51,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:51,489 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:52,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:53,430 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:54,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:55,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:56,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:57,359 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:48:57,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:58,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:48:59,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:00,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:01,431 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:02,364 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:49:02,437 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:03,015 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:49:03,048 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/config.yaml 2025-01-07 21:49:03,339 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: stop_status 2025-01-07 21:49:03,339 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: stop_status 2025-01-07 21:49:03,432 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:04,005 DEBUG SenderThread:2236235 [sender.py:send():379] send: exit 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:send_exit():586] handling exit code: 0 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:send_exit():588] handling runtime: 331 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:send_exit():594] send defer 2025-01-07 21:49:04,006 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:04,006 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-07 21:49:04,006 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-07 21:49:04,006 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 1 2025-01-07 21:49:04,007 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:04,007 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-07 21:49:04,007 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:04,007 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-07 21:49:04,007 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 2 2025-01-07 21:49:04,007 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:04,007 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-07 21:49:04,007 INFO HandlerThread:2236235 [system_monitor.py:finish():203] Stopping system monitor 2025-01-07 21:49:04,007 DEBUG SystemMonitor:2236235 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-07 21:49:04,007 DEBUG SystemMonitor:2236235 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-07 21:49:04,008 INFO HandlerThread:2236235 [interfaces.py:finish():200] Joined cpu monitor 2025-01-07 21:49:04,008 INFO HandlerThread:2236235 [interfaces.py:finish():200] Joined disk monitor 2025-01-07 21:49:04,048 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:49:04,061 INFO HandlerThread:2236235 [interfaces.py:finish():200] Joined gpu monitor 2025-01-07 21:49:04,062 INFO HandlerThread:2236235 [interfaces.py:finish():200] Joined memory monitor 2025-01-07 21:49:04,062 INFO HandlerThread:2236235 [interfaces.py:finish():200] Joined network monitor 2025-01-07 21:49:04,062 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:04,062 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-07 21:49:04,062 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 3 2025-01-07 21:49:04,062 DEBUG SenderThread:2236235 [sender.py:send():379] send: stats 2025-01-07 21:49:04,063 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:04,063 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-07 21:49:04,063 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:04,063 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-07 21:49:04,063 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 4 2025-01-07 21:49:04,064 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:04,064 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-07 21:49:06,049 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:49:10,775 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 21:49:10,775 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:49:10,775 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: keepalive 2025-01-07 21:49:10,776 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:10,776 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-07 21:49:10,776 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 5 2025-01-07 21:49:10,776 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 21:49:10,776 DEBUG SenderThread:2236235 [sender.py:send():379] send: history 2025-01-07 21:49:10,776 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: summary_record 2025-01-07 21:49:10,777 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:10,777 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-07 21:49:10,777 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:49:10,778 DEBUG SenderThread:2236235 [sender.py:send():379] send: summary 2025-01-07 21:49:10,778 INFO SenderThread:2236235 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-07 21:49:10,778 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:10,778 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-07 21:49:10,778 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 6 2025-01-07 21:49:10,778 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:10,778 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-07 21:49:10,778 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:10,779 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-07 21:49:10,779 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 7 2025-01-07 21:49:10,779 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: status_report 2025-01-07 21:49:10,779 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:10,779 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-07 21:49:10,779 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:10,779 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-07 21:49:11,007 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 21:49:11,051 INFO Thread-12 :2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:49:12,583 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 8 2025-01-07 21:49:12,583 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 21:49:12,584 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:12,584 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-07 21:49:12,584 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:12,584 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-07 21:49:12,584 INFO SenderThread:2236235 [job_builder.py:build():440] Attempting to build job artifact 2025-01-07 21:49:12,586 INFO SenderThread:2236235 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-07 21:49:12,588 INFO SenderThread:2236235 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-07 21:49:12,599 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 9 2025-01-07 21:49:12,600 DEBUG SenderThread:2236235 [sender.py:send():379] send: artifact 2025-01-07 21:49:12,602 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:12,602 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-07 21:49:12,788 INFO SenderThread:2236235 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjE0MDMwNQ==', 'versionIndex': 437}}} 2025-01-07 21:49:12,789 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:12,789 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-07 21:49:12,789 INFO SenderThread:2236235 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-07 21:49:13,007 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 21:49:13,052 INFO SenderThread:2236235 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:49:13,052 INFO SenderThread:2236235 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files 2025-01-07 21:49:13,053 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/config.yaml config.yaml 2025-01-07 21:49:13,053 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json wandb-summary.json 2025-01-07 21:49:13,053 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/requirements.txt requirements.txt 2025-01-07 21:49:13,053 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-metadata.json wandb-metadata.json 2025-01-07 21:49:13,053 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:49:13,055 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log output.log 2025-01-07 21:49:13,058 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/diff.patch diff.patch 2025-01-07 21:49:13,058 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-07 21:49:13,060 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 media/videos/eval/0_eval_0_1421fa3eb2dce107f426.mp4 2025-01-07 21:49:13,061 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 media/videos/eval/4_eval_4_658e49a8b695f808dd49.mp4 2025-01-07 21:49:13,061 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 media/videos/eval/3_eval_3_df2c07dbc3a06aaff176.mp4 2025-01-07 21:49:13,061 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 media/videos/eval/1_eval_1_cad41cf494b8d7558bdb.mp4 2025-01-07 21:49:13,061 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 media/videos/eval/2_eval_2_670babc1e5fba7a0258b.mp4 2025-01-07 21:49:13,062 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json media/table/eval/total_success_rate_ci_6_a21b0331f203f3ddd4b6.table.json 2025-01-07 21:49:13,062 INFO SenderThread:2236235 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json 2025-01-07 21:49:13,062 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 10 2025-01-07 21:49:13,062 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 21:49:13,062 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:13,062 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-07 21:49:13,062 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:13,062 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-07 21:49:13,062 INFO SenderThread:2236235 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 21:49:13,239 INFO wandb-upload_0:2236235 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/wandb-summary.json 2025-01-07 21:49:13,242 INFO wandb-upload_1:2236235 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/config.yaml 2025-01-07 21:49:13,262 INFO wandb-upload_2:2236235 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/requirements.txt 2025-01-07 21:49:13,275 INFO wandb-upload_4:2236235 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/files/output.log 2025-01-07 21:49:13,362 INFO wandb-upload_3:2236235 [upload_job.py:push():130] Uploaded file /tmp/tmpp7_3p_e5wandb/wke659n3-events.out.tfevents.1736304213.leonmkim-ROG-Strix-G15CS-G15CS.2235834.0 2025-01-07 21:49:13,562 INFO Thread-11 (_thread_body):2236235 [sender.py:transition_state():614] send defer: 11 2025-01-07 21:49:13,563 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:13,563 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-07 21:49:13,563 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:13,563 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-07 21:49:13,563 INFO SenderThread:2236235 [file_pusher.py:join():175] waiting for file pusher 2025-01-07 21:49:13,564 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 12 2025-01-07 21:49:13,564 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:13,564 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-07 21:49:13,565 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:13,565 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-07 21:49:13,565 INFO SenderThread:2236235 [file_stream.py:finish():601] file stream finish called 2025-01-07 21:49:13,760 INFO SenderThread:2236235 [file_stream.py:finish():605] file stream finish is done 2025-01-07 21:49:13,760 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 13 2025-01-07 21:49:13,760 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:13,760 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-07 21:49:13,760 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:13,760 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-07 21:49:13,760 INFO SenderThread:2236235 [sender.py:transition_state():614] send defer: 14 2025-01-07 21:49:13,760 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: defer 2025-01-07 21:49:13,760 DEBUG SenderThread:2236235 [sender.py:send():379] send: final 2025-01-07 21:49:13,760 INFO HandlerThread:2236235 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-07 21:49:13,760 DEBUG SenderThread:2236235 [sender.py:send():379] send: footer 2025-01-07 21:49:13,761 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: defer 2025-01-07 21:49:13,761 INFO SenderThread:2236235 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-07 21:49:13,761 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 21:49:13,761 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: server_info 2025-01-07 21:49:13,761 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-07 21:49:13,762 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: get_summary 2025-01-07 21:49:13,762 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-07 21:49:13,762 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-07 21:49:13,762 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 21:49:13,763 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: server_info 2025-01-07 21:49:13,812 DEBUG SenderThread:2236235 [sender.py:send_request():406] send_request: poll_exit 2025-01-07 21:49:13,813 INFO MainThread:2236235 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-07 21:49:13,813 INFO MainThread:2236235 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-07 21:49:13,813 INFO MainThread:2236235 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-07 21:49:13,814 DEBUG HandlerThread:2236235 [handler.py:handle_request():158] handle_request: shutdown 2025-01-07 21:49:13,814 INFO HandlerThread:2236235 [handler.py:finish():882] shutting down handler 2025-01-07 21:49:14,762 INFO WriterThread:2236235 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/b3d2fbi3/wandb/run-20250107_214332-b3d2fbi3/run-b3d2fbi3.wandb 2025-01-07 21:49:14,814 INFO SenderThread:2236235 [sender.py:finish():1615] shutting down sender 2025-01-07 21:49:14,814 INFO SenderThread:2236235 [file_pusher.py:finish():169] shutting down file pusher 2025-01-07 21:49:14,814 INFO SenderThread:2236235 [file_pusher.py:join():175] waiting for file pusher