2025-01-21 21:13:39,120 INFO StreamThr :2647496 [internal.py:wandb_internal():85] W&B internal server running at pid: 2647496, started at: 2025-01-21 21:13:39.120468 2025-01-21 21:13:39,123 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status 2025-01-21 21:13:39,133 INFO WriterThread:2647496 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/run-2o1us5j0.wandb 2025-01-21 21:13:39,133 DEBUG SenderThread:2647496 [sender.py:send():379] send: header 2025-01-21 21:13:39,141 DEBUG SenderThread:2647496 [sender.py:send():379] send: run 2025-01-21 21:13:39,450 INFO SenderThread:2647496 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files 2025-01-21 21:13:39,450 INFO SenderThread:2647496 [sender.py:_start_run_threads():1188] run started: 2o1us5j0 with start time 1737512019.120725 2025-01-21 21:13:39,462 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: check_version 2025-01-21 21:13:39,462 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: check_version 2025-01-21 21:13:39,541 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: run_start 2025-01-21 21:13:39,546 DEBUG HandlerThread:2647496 [system_info.py:__init__():26] System info init 2025-01-21 21:13:39,546 DEBUG HandlerThread:2647496 [system_info.py:__init__():41] System info init done 2025-01-21 21:13:39,546 INFO HandlerThread:2647496 [system_monitor.py:start():194] Starting system monitor 2025-01-21 21:13:39,546 INFO SystemMonitor:2647496 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-21 21:13:39,547 INFO HandlerThread:2647496 [system_monitor.py:probe():214] Collecting system info 2025-01-21 21:13:39,547 INFO SystemMonitor:2647496 [interfaces.py:start():188] Started cpu monitoring 2025-01-21 21:13:39,548 INFO SystemMonitor:2647496 [interfaces.py:start():188] Started disk monitoring 2025-01-21 21:13:39,549 INFO SystemMonitor:2647496 [interfaces.py:start():188] Started gpu monitoring 2025-01-21 21:13:39,549 INFO SystemMonitor:2647496 [interfaces.py:start():188] Started memory monitoring 2025-01-21 21:13:39,549 INFO SystemMonitor:2647496 [interfaces.py:start():188] Started network monitoring 2025-01-21 21:13:39,560 DEBUG HandlerThread:2647496 [system_info.py:probe():152] Probing system 2025-01-21 21:13:39,561 DEBUG HandlerThread:2647496 [system_info.py:_probe_git():137] Probing git 2025-01-21 21:13:39,566 DEBUG HandlerThread:2647496 [system_info.py:_probe_git():145] Probing git done 2025-01-21 21:13:39,566 DEBUG HandlerThread:2647496 [system_info.py:probe():200] Probing system done 2025-01-21 21:13:39,566 DEBUG HandlerThread:2647496 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-22T02:13:39.560261', 'startedAt': '2025-01-22T02:13:39.112325', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'dc9da9fa0ef2996c7fec8d105e7933e728cd952f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.98925, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4491.036, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 350.6014938354492}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-21 21:13:39,566 INFO HandlerThread:2647496 [system_monitor.py:probe():224] Finished collecting system info 2025-01-21 21:13:39,566 INFO HandlerThread:2647496 [system_monitor.py:probe():227] Publishing system info 2025-01-21 21:13:39,566 DEBUG HandlerThread:2647496 [system_info.py:_save_code():44] Saving code 2025-01-21 21:13:39,567 DEBUG HandlerThread:2647496 [system_info.py:_save_code():67] Saving code done 2025-01-21 21:13:39,567 DEBUG HandlerThread:2647496 [system_info.py:_save_patches():84] Saving git patches 2025-01-21 21:13:39,629 DEBUG HandlerThread:2647496 [system_info.py:_save_patches():126] Saving git patches done 2025-01-21 21:13:39,630 INFO HandlerThread:2647496 [system_monitor.py:probe():229] Finished publishing system info 2025-01-21 21:13:39,633 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:13:39,633 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-21 21:13:39,633 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-21 21:13:39,633 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-21 21:13:39,897 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: python_packages 2025-01-21 21:13:39,897 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: python_packages 2025-01-21 21:13:39,899 DEBUG SenderThread:2647496 [sender.py:send():379] send: telemetry 2025-01-21 21:13:39,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:13:39,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:39,900 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:13:39,902 INFO HandlerThread:2647496 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/tb" save: true } control { relay_id: "07530a4f366b4424bd90a669e2ceac72" } _info { stream_id: "2o1us5j0" } 2025-01-21 21:13:39,902 DEBUG HandlerThread:2647496 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-21 21:13:39,926 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/fxfagu85-diff.patch 2025-01-21 21:13:39,928 INFO wandb-upload_1:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/yvoo1sr9-code/FISH/eval_robot.py 2025-01-21 21:13:39,938 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/rtiscy5w-wandb-metadata.json 2025-01-21 21:13:40,029 DEBUG SenderThread:2647496 [sender.py:send():379] send: tbrecord 2025-01-21 21:13:40,029 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:13:40,029 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 with policy live 2025-01-21 21:13:40,212 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/6be263eg-events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/diff.patch 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-metadata.json 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/requirements.txt 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/code/FISH/eval_robot.py 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/code 2025-01-21 21:13:40,452 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/code/FISH 2025-01-21 21:13:40,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:41,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:42,451 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:13:42,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:43,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:44,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:13:44,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:45,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:46,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:47,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:48,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:49,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:13:49,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:50,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:51,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:52,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:53,110 DEBUG SenderThread:2647496 [sender.py:send():379] send: config 2025-01-21 21:13:53,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:54,454 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:13:54,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:13:54,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:13:54,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:54,971 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:13:55,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:56,455 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:13:56,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:57,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:58,455 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:13:58,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:13:59,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:00,456 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:00,547 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:00,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:01,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:02,457 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:02,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:03,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:04,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:05,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:05,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:06,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:07,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:08,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:09,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:14:09,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:14:09,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:10,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:11,004 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:11,459 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/config.yaml 2025-01-21 21:14:11,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:12,459 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:12,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:13,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:14,460 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:14,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:15,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:16,327 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:16,461 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:16,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:17,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:18,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:19,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:20,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:21,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:21,904 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:22,904 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:23,904 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:24,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:14:24,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:14:24,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:25,904 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:26,904 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:26,988 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:27,905 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:28,905 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:29,905 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:30,905 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:31,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:32,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:32,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:33,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:34,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:35,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:36,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:37,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:38,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:38,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:39,550 DEBUG SystemMonitor:2647496 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-21 21:14:39,550 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:14:39,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:14:39,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:14:39,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:40,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:41,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:42,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:43,049 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:14:43,050 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:14:43,050 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 with policy now 2025-01-21 21:14:43,050 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:14:43,051 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:14:43,051 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:14:43,299 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/nhde7sj8-media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 2025-01-21 21:14:43,469 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:14:43,469 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 2025-01-21 21:14:43,469 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos 2025-01-21 21:14:43,469 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval 2025-01-21 21:14:43,470 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media 2025-01-21 21:14:43,767 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/rhq817hg-media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 2025-01-21 21:14:43,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:44,051 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:44,469 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:44,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:45,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:46,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:47,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:48,471 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:48,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:49,346 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:49,908 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:50,471 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:14:50,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:51,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:52,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:53,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:54,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:14:54,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:14:54,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:55,033 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:14:55,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:56,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:57,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:58,909 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:14:59,911 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:00,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:00,910 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:01,475 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:15:01,733 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/77lcrimn-events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:15:01,910 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:02,475 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:02,911 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:03,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:04,476 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:04,945 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:05,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:06,156 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:06,477 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:06,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:07,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:08,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:09,551 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:15:09,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:15:09,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:15:09,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:10,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:11,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:12,035 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:12,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:13,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:14,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:15,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:16,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:17,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:17,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:18,913 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:19,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:20,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:21,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:22,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:22,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:23,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:24,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:15:24,899 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:15:24,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:25,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:26,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:27,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:27,990 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:28,914 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:29,915 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:30,915 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:31,915 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:32,915 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:33,107 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:33,153 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:15:33,154 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:15:33,154 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 with policy now 2025-01-21 21:15:33,154 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:15:33,154 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:15:33,157 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:15:33,395 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/16b80rlp-media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 2025-01-21 21:15:33,485 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:15:33,486 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 2025-01-21 21:15:33,486 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval 2025-01-21 21:15:33,785 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/z7rklwx6-media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 2025-01-21 21:15:33,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:34,486 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:34,919 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:35,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:36,919 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:37,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:38,487 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:38,789 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:38,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:39,551 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:15:39,898 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:15:39,898 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:15:39,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:40,488 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:40,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:41,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:42,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:43,489 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:15:43,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:44,333 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:44,339 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:15:44,339 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:15:44,339 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:15:44,489 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:15:44,490 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:44,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:45,918 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:46,490 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:46,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:47,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:48,491 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:15:48,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:49,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:49,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:50,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:51,920 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:52,921 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:53,921 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:54,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:15:54,899 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:15:54,945 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:54,973 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:15:55,927 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:56,927 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:57,927 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:58,927 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:15:59,928 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:00,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:00,929 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:01,929 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:02,496 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:02,930 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:03,930 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:04,930 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:05,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:05,930 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:06,931 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:07,931 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:08,497 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:08,931 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:09,552 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:16:09,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:16:09,899 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:16:09,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:10,936 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:11,011 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:11,937 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:12,937 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:13,937 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:14,946 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:15,329 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:16:15,329 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 with policy now 2025-01-21 21:16:15,334 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:16:15,334 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:16:15,334 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:16:15,334 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:16:15,500 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:16:15,500 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 2025-01-21 21:16:15,500 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval 2025-01-21 21:16:15,566 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/bhs56rj6-media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 2025-01-21 21:16:15,839 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/xtywq2ny-media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 2025-01-21 21:16:15,946 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:16,373 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:16,500 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:16,946 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:17,942 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:18,942 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:19,945 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:20,501 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:20,945 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:21,767 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:21,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:22,945 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:23,948 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:24,503 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:24,902 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:16:24,902 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:16:24,953 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:25,156 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:16:25,156 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:16:25,156 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:16:25,503 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:16:25,503 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:16:25,767 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/eblg8jeh-events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:16:25,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:26,503 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:26,869 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:26,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:27,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:28,504 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:28,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:29,948 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:30,505 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:30,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:31,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:32,107 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:32,506 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:32,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:33,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:34,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:35,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:36,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:37,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:37,949 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:38,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:39,552 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:16:39,899 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:16:39,900 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:16:39,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:40,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:41,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:42,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:42,981 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:43,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:44,950 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:45,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:46,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:47,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:48,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:48,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:49,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:50,951 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:51,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:52,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:53,907 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:53,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:54,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:16:54,900 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:16:54,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:55,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:56,953 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:57,954 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:58,050 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:16:58,050 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 with policy now 2025-01-21 21:16:58,053 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:16:58,053 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:16:58,053 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:16:58,053 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:16:58,299 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/vpsz4ygw-media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 2025-01-21 21:16:58,515 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:16:58,515 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:16:58,515 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 2025-01-21 21:16:58,515 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval 2025-01-21 21:16:58,801 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/eqn0vb4m-media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 2025-01-21 21:16:58,955 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:16:59,080 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:16:59,954 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:00,515 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:00,958 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:01,966 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:02,516 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:02,962 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:03,960 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:04,470 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:04,954 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:05,956 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:06,517 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:06,960 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:07,964 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:08,966 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:09,553 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:17:09,553 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:09,906 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:17:09,907 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:17:09,967 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:10,965 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:11,973 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:12,973 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:13,971 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:14,211 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:17:14,211 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:14,212 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:14,520 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:17:14,520 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:17:14,912 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:14,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:15,971 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:16,521 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:16,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:18,017 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:18,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:19,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:20,522 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:20,911 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:20,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:21,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:22,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:23,968 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:24,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:17:24,900 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:17:24,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:25,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:26,061 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:26,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:27,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:28,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:29,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:30,525 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:30,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:31,911 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:31,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:32,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:33,969 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:34,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:35,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:36,527 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:36,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:36,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:37,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:38,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:39,553 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:17:39,900 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:17:39,900 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:17:39,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:40,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:41,970 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:42,041 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:42,530 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:42,971 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:43,971 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:44,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:45,975 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:46,326 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:17:46,327 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 with policy now 2025-01-21 21:17:46,333 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:17:46,333 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:17:46,333 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:46,334 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:46,531 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:17:46,531 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:46,531 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 2025-01-21 21:17:46,531 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval 2025-01-21 21:17:46,555 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/i1jwc2vc-media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 2025-01-21 21:17:46,824 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/mrdlrkek-media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 2025-01-21 21:17:46,978 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:47,350 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:47,976 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:48,532 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:48,978 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:50,005 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:50,983 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:51,982 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:52,533 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:52,807 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:52,982 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:53,980 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:54,534 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:54,926 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:17:54,926 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:17:54,994 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:55,992 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:56,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:57,535 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:17:57,798 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/c9psj602-events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:17:57,995 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:58,267 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:17:58,267 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:58,267 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:17:58,268 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:58,342 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-21 21:17:58,342 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: log_artifact 2025-01-21 21:17:58,535 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:17:58,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:17:58,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-21 21:17:58,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media 2025-01-21 21:17:58,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table 2025-01-21 21:17:58,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval 2025-01-21 21:17:58,549 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:17:58,730 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/3cpmjebx-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-21 21:17:58,825 INFO wandb-upload_0:2647496 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp3ejflfr6 2025-01-21 21:17:58,941 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-21 21:17:58,952 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 21:17:58,984 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:17:59,232 INFO SenderThread:2647496 [sender.py:send_request_log_artifact():1525] logged artifact run-2o1us5j0-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY1ODIxNQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5Mzc4MA==', 'latestArtifact': None}} 2025-01-21 21:17:59,233 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:17:59,233 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now 2025-01-21 21:17:59,233 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:17:59,233 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,234 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,234 DEBUG SenderThread:2647496 [sender.py:send():379] send: telemetry 2025-01-21 21:17:59,234 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,234 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,234 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,235 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,235 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,237 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,237 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,237 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,237 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,238 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,238 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,238 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,238 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,239 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,239 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,239 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,239 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,240 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,240 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,240 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,240 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:17:59,240 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:17:59,241 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: log_artifact 2025-01-21 21:17:59,495 INFO wandb-upload_1:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/mvi0bfk2-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-21 21:17:59,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:17:59,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:17:59,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json 2025-01-21 21:17:59,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval 2025-01-21 21:17:59,587 INFO wandb-upload_0:2647496 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpjo3psi_s 2025-01-21 21:17:59,718 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/u5jhwjet-media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json 2025-01-21 21:17:59,984 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:00,109 INFO SenderThread:2647496 [sender.py:send_request_log_artifact():1525] logged artifact run-2o1us5j0-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY1ODIyOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5Mzc4Ng==', 'latestArtifact': None}} 2025-01-21 21:18:00,109 DEBUG SenderThread:2647496 [sender.py:send():379] send: files 2025-01-21 21:18:00,109 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json with policy now 2025-01-21 21:18:00,109 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:18:00,109 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:18:00,110 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:18:00,366 INFO wandb-upload_1:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/hkyl84mn-media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json 2025-01-21 21:18:00,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:18:00,536 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:18:00,984 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:01,984 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:02,984 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:03,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:04,111 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:18:04,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:05,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:06,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:07,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:08,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:09,554 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:18:09,555 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:18:09,901 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 21:18:09,901 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: stop_status 2025-01-21 21:18:09,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:10,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:11,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:12,985 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:13,986 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:14,917 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:18:14,986 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:15,986 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:16,901 DEBUG SenderThread:2647496 [sender.py:send():379] send: exit 2025-01-21 21:18:16,902 INFO SenderThread:2647496 [sender.py:send_exit():586] handling exit code: 0 2025-01-21 21:18:16,902 INFO SenderThread:2647496 [sender.py:send_exit():588] handling runtime: 277 2025-01-21 21:18:16,902 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:18:16,903 INFO SenderThread:2647496 [sender.py:send_exit():594] send defer 2025-01-21 21:18:16,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:16,903 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-21 21:18:16,903 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:16,903 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-21 21:18:16,903 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 1 2025-01-21 21:18:16,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:16,903 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-21 21:18:16,903 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:16,903 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-21 21:18:16,903 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 2 2025-01-21 21:18:16,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:16,904 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-21 21:18:16,904 INFO HandlerThread:2647496 [system_monitor.py:finish():203] Stopping system monitor 2025-01-21 21:18:16,904 DEBUG SystemMonitor:2647496 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-21 21:18:16,904 DEBUG SystemMonitor:2647496 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-21 21:18:16,904 INFO HandlerThread:2647496 [interfaces.py:finish():200] Joined cpu monitor 2025-01-21 21:18:16,905 INFO HandlerThread:2647496 [interfaces.py:finish():200] Joined disk monitor 2025-01-21 21:18:16,957 INFO HandlerThread:2647496 [interfaces.py:finish():200] Joined gpu monitor 2025-01-21 21:18:16,957 INFO HandlerThread:2647496 [interfaces.py:finish():200] Joined memory monitor 2025-01-21 21:18:16,958 INFO HandlerThread:2647496 [interfaces.py:finish():200] Joined network monitor 2025-01-21 21:18:16,958 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:16,958 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-21 21:18:16,958 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 3 2025-01-21 21:18:16,958 DEBUG SenderThread:2647496 [sender.py:send():379] send: stats 2025-01-21 21:18:16,959 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:16,959 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-21 21:18:16,959 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:16,959 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-21 21:18:16,959 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 4 2025-01-21 21:18:16,960 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:16,960 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-21 21:18:17,543 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:18:18,544 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:18:23,274 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:23,274 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:18:23,274 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 21:18:23,276 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:23,276 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-21 21:18:23,276 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 5 2025-01-21 21:18:23,276 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:23,276 DEBUG SenderThread:2647496 [sender.py:send():379] send: history 2025-01-21 21:18:23,276 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: summary_record 2025-01-21 21:18:23,276 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:23,277 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-21 21:18:23,278 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:18:23,278 DEBUG SenderThread:2647496 [sender.py:send():379] send: summary 2025-01-21 21:18:23,282 INFO SenderThread:2647496 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 21:18:23,282 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:23,282 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-21 21:18:23,282 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 6 2025-01-21 21:18:23,282 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:23,282 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-21 21:18:23,282 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:23,282 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-21 21:18:23,286 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 21:18:23,549 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 7 2025-01-21 21:18:23,549 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:23,550 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-21 21:18:23,550 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:23,550 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-21 21:18:23,550 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:18:23,550 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/config.yaml 2025-01-21 21:18:23,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:24,357 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 8 2025-01-21 21:18:24,357 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:24,357 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:24,358 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-21 21:18:24,358 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:24,358 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-21 21:18:24,358 INFO SenderThread:2647496 [job_builder.py:build():440] Attempting to build job artifact 2025-01-21 21:18:24,360 INFO SenderThread:2647496 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-21 21:18:24,363 INFO SenderThread:2647496 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-21 21:18:24,373 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 9 2025-01-21 21:18:24,373 DEBUG SenderThread:2647496 [sender.py:send():379] send: artifact 2025-01-21 21:18:24,373 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:24,376 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-21 21:18:24,551 INFO Thread-12 :2647496 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:18:24,565 INFO SenderThread:2647496 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1NzYyNTUzOQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1NzYyNTUzOQ==', 'versionIndex': 454}}} 2025-01-21 21:18:24,565 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:24,565 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-21 21:18:24,565 INFO SenderThread:2647496 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-21 21:18:24,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/config.yaml config.yaml 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json wandb-summary.json 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/requirements.txt requirements.txt 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-metadata.json wandb-metadata.json 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:18:25,552 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log output.log 2025-01-21 21:18:25,557 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/diff.patch diff.patch 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 media/videos/eval/3_eval_3_d1146ecf321ca00c51fe.mp4 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 media/videos/eval/0_eval_0_7b23de2781f443affd89.mp4 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 media/videos/eval/1_eval_1_27ab141d29c0759686c9.mp4 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 media/videos/eval/2_eval_2_02193a80ccc515d7b53d.mp4 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 media/videos/eval/4_eval_4_99c129977338f91caa37.mp4 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json media/table/eval/total_success_rate_ci_6_8e64a9a966331685f358.table.json 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json 2025-01-21 21:18:25,558 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 10 2025-01-21 21:18:25,559 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:25,559 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:25,559 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-21 21:18:25,559 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:25,559 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-21 21:18:25,559 INFO SenderThread:2647496 [file_pusher.py:finish():169] shutting down file pusher 2025-01-21 21:18:25,746 INFO wandb-upload_2:2647496 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/config.yaml 2025-01-21 21:18:25,758 INFO wandb-upload_1:2647496 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/requirements.txt 2025-01-21 21:18:25,774 INFO wandb-upload_4:2647496 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/output.log 2025-01-21 21:18:25,775 INFO wandb-upload_0:2647496 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/files/wandb-summary.json 2025-01-21 21:18:25,884 INFO wandb-upload_3:2647496 [upload_job.py:push():130] Uploaded file /tmp/tmpi7mqcor2wandb/k7rh9v1y-events.out.tfevents.1737512019.leonmkim-ROG-Strix-G15CS-G15CS.2647330.0 2025-01-21 21:18:25,903 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:25,903 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:26,085 INFO Thread-11 (_thread_body):2647496 [sender.py:transition_state():614] send defer: 11 2025-01-21 21:18:26,085 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:26,085 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-21 21:18:26,085 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:26,085 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-21 21:18:26,085 INFO SenderThread:2647496 [file_pusher.py:join():175] waiting for file pusher 2025-01-21 21:18:26,087 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 12 2025-01-21 21:18:26,087 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:26,087 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-21 21:18:26,087 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:26,087 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-21 21:18:26,087 INFO SenderThread:2647496 [file_stream.py:finish():601] file stream finish called 2025-01-21 21:18:26,295 INFO SenderThread:2647496 [file_stream.py:finish():605] file stream finish is done 2025-01-21 21:18:26,295 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 13 2025-01-21 21:18:26,296 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:26,296 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-21 21:18:26,296 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:26,296 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-21 21:18:26,296 INFO SenderThread:2647496 [sender.py:transition_state():614] send defer: 14 2025-01-21 21:18:26,296 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: defer 2025-01-21 21:18:26,296 INFO HandlerThread:2647496 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-21 21:18:26,296 DEBUG SenderThread:2647496 [sender.py:send():379] send: final 2025-01-21 21:18:26,296 DEBUG SenderThread:2647496 [sender.py:send():379] send: footer 2025-01-21 21:18:26,297 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: defer 2025-01-21 21:18:26,297 INFO SenderThread:2647496 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-21 21:18:26,297 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:26,297 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: server_info 2025-01-21 21:18:26,297 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:26,298 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 21:18:26,298 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: server_info 2025-01-21 21:18:26,298 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: get_summary 2025-01-21 21:18:26,300 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-21 21:18:26,301 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 21:18:26,365 DEBUG SenderThread:2647496 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 21:18:26,366 INFO MainThread:2647496 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-21 21:18:26,366 INFO MainThread:2647496 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-21 21:18:26,366 INFO MainThread:2647496 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-21 21:18:26,367 DEBUG HandlerThread:2647496 [handler.py:handle_request():158] handle_request: shutdown 2025-01-21 21:18:26,367 INFO HandlerThread:2647496 [handler.py:finish():882] shutting down handler 2025-01-21 21:18:27,298 INFO WriterThread:2647496 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/2o1us5j0/wandb/run-20250121_211339-2o1us5j0/run-2o1us5j0.wandb 2025-01-21 21:18:27,366 INFO SenderThread:2647496 [sender.py:finish():1615] shutting down sender 2025-01-21 21:18:27,366 INFO SenderThread:2647496 [file_pusher.py:finish():169] shutting down file pusher 2025-01-21 21:18:27,366 INFO SenderThread:2647496 [file_pusher.py:join():175] waiting for file pusher