2024-12-31 21:03:23,435 INFO StreamThr :1740211 [internal.py:wandb_internal():85] W&B internal server running at pid: 1740211, started at: 2024-12-31 21:03:23.435568 2024-12-31 21:03:23,436 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status 2024-12-31 21:03:23,449 INFO WriterThread:1740211 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/run-nkdxms1x.wandb 2024-12-31 21:03:23,450 DEBUG SenderThread:1740211 [sender.py:send():379] send: header 2024-12-31 21:03:23,459 DEBUG SenderThread:1740211 [sender.py:send():379] send: run 2024-12-31 21:03:23,694 INFO SenderThread:1740211 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files 2024-12-31 21:03:23,694 INFO SenderThread:1740211 [sender.py:_start_run_threads():1188] run started: nkdxms1x with start time 1735697003.437517 2024-12-31 21:03:23,711 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: check_version 2024-12-31 21:03:23,711 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: check_version 2024-12-31 21:03:23,755 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: run_start 2024-12-31 21:03:23,760 DEBUG HandlerThread:1740211 [system_info.py:__init__():26] System info init 2024-12-31 21:03:23,760 DEBUG HandlerThread:1740211 [system_info.py:__init__():41] System info init done 2024-12-31 21:03:23,760 INFO HandlerThread:1740211 [system_monitor.py:start():194] Starting system monitor 2024-12-31 21:03:23,760 INFO SystemMonitor:1740211 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-31 21:03:23,761 INFO HandlerThread:1740211 [system_monitor.py:probe():214] Collecting system info 2024-12-31 21:03:23,761 INFO SystemMonitor:1740211 [interfaces.py:start():188] Started cpu monitoring 2024-12-31 21:03:23,761 INFO SystemMonitor:1740211 [interfaces.py:start():188] Started disk monitoring 2024-12-31 21:03:23,761 INFO SystemMonitor:1740211 [interfaces.py:start():188] Started gpu monitoring 2024-12-31 21:03:23,762 INFO SystemMonitor:1740211 [interfaces.py:start():188] Started memory monitoring 2024-12-31 21:03:23,762 INFO SystemMonitor:1740211 [interfaces.py:start():188] Started network monitoring 2024-12-31 21:03:23,773 DEBUG HandlerThread:1740211 [system_info.py:probe():152] Probing system 2024-12-31 21:03:23,774 DEBUG HandlerThread:1740211 [system_info.py:_probe_git():137] Probing git 2024-12-31 21:03:23,779 DEBUG HandlerThread:1740211 [system_info.py:_probe_git():145] Probing git done 2024-12-31 21:03:23,779 DEBUG HandlerThread:1740211 [system_info.py:probe():200] Probing system done 2024-12-31 21:03:23,779 DEBUG HandlerThread:1740211 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T02:03:23.773340', 'startedAt': '2025-01-01T02:03:23.425372', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '284407f87cd673789a10dbd8e4d4f41dcad30ded'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3750.004875, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.995, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 495.41462326049805}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2024-12-31 21:03:23,779 INFO HandlerThread:1740211 [system_monitor.py:probe():224] Finished collecting system info 2024-12-31 21:03:23,779 INFO HandlerThread:1740211 [system_monitor.py:probe():227] Publishing system info 2024-12-31 21:03:23,779 DEBUG HandlerThread:1740211 [system_info.py:_save_code():44] Saving code 2024-12-31 21:03:23,780 DEBUG HandlerThread:1740211 [system_info.py:_save_code():67] Saving code done 2024-12-31 21:03:23,781 DEBUG HandlerThread:1740211 [system_info.py:_save_patches():84] Saving git patches 2024-12-31 21:03:23,842 DEBUG HandlerThread:1740211 [system_info.py:_save_patches():126] Saving git patches done 2024-12-31 21:03:23,842 INFO HandlerThread:1740211 [system_monitor.py:probe():229] Finished publishing system info 2024-12-31 21:03:23,845 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:03:23,845 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-31 21:03:23,846 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2024-12-31 21:03:23,846 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file diff.patch with policy now 2024-12-31 21:03:24,088 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: python_packages 2024-12-31 21:03:24,088 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: python_packages 2024-12-31 21:03:24,089 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/q79k8vpc-code/FISH/eval_robot.py 2024-12-31 21:03:24,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:03:24,090 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:03:24,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:24,091 INFO HandlerThread:1740211 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/tb" save: true } control { relay_id: "ee70b23f69f642a48cfae1d64e8e8ef4" } _info { stream_id: "nkdxms1x" } 2024-12-31 21:03:24,091 DEBUG HandlerThread:1740211 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2024-12-31 21:03:24,094 INFO wandb-upload_0:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/e8mji6dz-wandb-metadata.json 2024-12-31 21:03:24,107 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/ti5udqhq-diff.patch 2024-12-31 21:03:24,159 DEBUG SenderThread:1740211 [sender.py:send():379] send: telemetry 2024-12-31 21:03:24,159 DEBUG SenderThread:1740211 [sender.py:send():379] send: tbrecord 2024-12-31 21:03:24,160 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:03:24,160 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 with policy live 2024-12-31 21:03:24,345 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/xguvxaej-events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-metadata.json 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/code/FISH/eval_robot.py 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/requirements.txt 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/diff.patch 2024-12-31 21:03:24,696 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/code/FISH 2024-12-31 21:03:24,697 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/code 2024-12-31 21:03:25,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:26,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:26,695 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:27,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:28,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:29,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:29,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:29,760 DEBUG SenderThread:1740211 [sender.py:send():379] send: config 2024-12-31 21:03:30,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:31,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:32,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:32,697 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:33,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:34,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:34,495 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:34,697 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:35,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:36,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:36,698 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:37,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:38,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:38,699 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:39,088 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:03:39,089 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:03:39,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:40,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:40,169 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:41,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:42,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:43,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:44,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:45,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:45,988 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:46,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:46,701 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:47,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:48,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:48,701 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:49,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:50,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:50,702 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:51,015 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:51,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:52,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:52,704 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:53,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:54,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:03:54,089 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:03:54,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:54,704 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:55,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:56,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:56,178 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:03:56,705 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/config.yaml 2024-12-31 21:03:56,705 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:57,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:58,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:03:58,706 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:03:59,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:00,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:00,706 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:01,093 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:01,508 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:02,093 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:02,707 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:03,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:04,093 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:04,559 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:04:04,559 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 with policy now 2024-12-31 21:04:04,561 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:04:04,562 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:04:04,562 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:04:04,562 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:04:04,707 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:04,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:04:04,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 2024-12-31 21:04:04,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval 2024-12-31 21:04:04,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media 2024-12-31 21:04:04,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos 2024-12-31 21:04:04,797 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/uj3y4dh9-media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 2024-12-31 21:04:05,052 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/cgpichjt-media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 2024-12-31 21:04:05,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:06,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:06,573 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:06,708 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:07,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:08,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:09,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:04:09,091 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:04:09,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:10,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:10,709 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:11,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:12,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:12,533 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:12,710 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:13,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:14,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:15,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:16,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:17,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:18,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:18,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:19,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:20,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:21,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:22,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:23,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:23,762 DEBUG SystemMonitor:1740211 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-31 21:04:23,763 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:04:23,764 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:24,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:04:24,089 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:04:24,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:25,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:26,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:27,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:28,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:29,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:29,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:30,096 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:31,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:32,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:33,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:34,091 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:34,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:35,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:36,098 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:36,718 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:04:37,005 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/xgxz2wki-events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:04:37,098 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:38,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:38,719 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:39,089 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:04:39,089 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:04:39,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:39,237 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:40,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:41,109 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:42,109 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:42,720 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:43,109 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:44,109 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:44,407 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:44,721 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:45,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:46,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:46,722 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:47,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:48,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:48,723 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:49,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:49,414 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:50,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:50,723 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:51,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:52,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:52,724 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:53,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:53,763 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:04:54,092 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:04:54,092 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:04:54,118 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:54,327 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:04:54,327 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 with policy now 2024-12-31 21:04:54,331 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:04:54,332 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:04:54,332 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:04:54,332 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:04:54,567 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/m9tsz8o4-media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 2024-12-31 21:04:54,725 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:04:54,725 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:54,726 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 2024-12-31 21:04:54,726 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval 2024-12-31 21:04:54,973 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/apzt3djp-media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 2024-12-31 21:04:55,117 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:55,354 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:04:56,115 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:56,726 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:57,116 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:58,120 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:04:58,726 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:04:59,126 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:00,127 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:00,627 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:00,727 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:01,125 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:02,125 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:02,728 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:03,125 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:04,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:05,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:06,093 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:06,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:07,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:08,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:09,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:05:09,090 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:05:09,133 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:10,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:11,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:11,211 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:12,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:13,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:14,128 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:15,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:16,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:17,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:17,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:18,130 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:19,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:19,501 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:05:19,501 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:05:19,502 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:05:19,734 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:05:19,734 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:05:20,130 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:20,734 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:21,129 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:22,130 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:22,269 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:22,735 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:23,144 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:23,764 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:05:24,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:05:24,090 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:05:24,135 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:24,736 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:25,135 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:26,135 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:26,737 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:27,136 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:27,441 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:28,136 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:28,738 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:29,136 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:30,136 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:30,738 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:31,136 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:32,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:32,448 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:32,739 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:33,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:34,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:34,740 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:35,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:36,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:36,741 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:37,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:37,469 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:38,137 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:38,741 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:39,090 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:05:39,090 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:05:39,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:40,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:40,742 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:41,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:42,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:42,642 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:42,743 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:43,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:44,138 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:44,744 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:45,139 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:46,139 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:46,744 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:47,139 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:47,650 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:48,139 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:48,745 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:49,139 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:50,142 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:50,746 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:50,924 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:05:50,924 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 with policy now 2024-12-31 21:05:50,924 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:05:50,925 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:05:50,925 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:05:50,925 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:05:51,143 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:51,174 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/ueb0i9jh-media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 2024-12-31 21:05:51,746 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:05:51,746 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 2024-12-31 21:05:51,747 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval 2024-12-31 21:05:52,054 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/1ppz161i-media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 2024-12-31 21:05:52,141 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:52,746 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:52,959 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:53,146 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:53,764 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:05:54,102 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:05:54,102 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:05:54,149 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:55,147 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:56,145 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:56,748 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:57,145 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:58,146 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:58,393 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:05:58,749 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:05:59,148 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:05:59,329 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:05:59,330 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:05:59,330 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:05:59,749 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:05:59,749 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:05:59,980 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/iaw5gjma-events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:06:00,149 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:00,749 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:01,151 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:02,151 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:03,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:03,471 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:04,165 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:04,751 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:05,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:06,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:06,752 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:07,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:08,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:08,502 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:08,752 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:09,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:06:09,095 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:06:09,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:10,166 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:10,753 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:11,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:12,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:12,754 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:13,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:13,697 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:14,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:14,755 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:15,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:16,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:16,755 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:17,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:18,167 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:18,698 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:18,756 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:19,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:20,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:20,757 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:21,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:22,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:22,758 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:23,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:23,702 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:23,765 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:06:24,094 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:06:24,095 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:06:24,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:24,758 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:25,168 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:26,169 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:26,759 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:27,169 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:28,169 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:28,760 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:28,906 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:29,169 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:30,173 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:30,760 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:31,152 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:06:31,153 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_a0e47918912079386856.mp4 with policy now 2024-12-31 21:06:31,156 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:06:31,156 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:06:31,156 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:06:31,157 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:06:31,171 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:31,412 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/tjltem0s-media/videos/eval/3_eval_3_a0e47918912079386856.mp4 2024-12-31 21:06:31,761 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:06:31,761 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/3_eval_3_a0e47918912079386856.mp4 2024-12-31 21:06:31,761 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval 2024-12-31 21:06:32,054 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/a1eck4xq-media/videos/eval/3_eval_3_a0e47918912079386856.mp4 2024-12-31 21:06:32,185 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:32,761 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:33,180 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:34,107 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:34,178 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:35,180 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:36,185 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:36,763 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:37,186 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:38,181 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:38,763 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:39,098 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:06:39,098 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:06:39,189 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:39,210 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:40,183 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:41,186 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:42,188 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:43,186 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:44,190 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:45,106 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:45,191 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:46,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:47,190 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:47,766 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:06:48,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:48,388 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:06:48,388 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:06:48,389 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:06:48,767 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:06:48,767 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:49,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:50,122 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:50,197 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:50,767 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:51,196 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:52,190 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:52,768 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:53,191 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:53,769 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:06:54,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:06:54,096 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:06:54,191 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:54,770 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:55,133 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:06:55,191 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:56,192 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:56,770 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:57,192 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:58,192 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:06:58,771 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:06:59,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:00,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:00,309 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:00,772 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:01,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:02,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:02,773 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:03,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:04,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:04,773 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:05,193 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:05,326 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:06,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:06,774 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:07,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:08,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:08,775 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:09,095 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:07:09,096 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:07:09,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:10,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:10,516 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:10,776 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:11,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:12,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:12,777 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:13,194 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:14,195 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:14,777 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:15,195 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:15,719 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:16,195 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:16,778 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:17,195 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:18,195 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:18,779 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:19,201 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:19,953 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:07:19,953 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 with policy now 2024-12-31 21:07:19,956 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:07:19,956 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:07:19,956 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:19,957 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:20,199 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:20,214 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/n4g10zrq-media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 2024-12-31 21:07:20,780 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:20,780 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:20,780 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 2024-12-31 21:07:20,780 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval 2024-12-31 21:07:20,976 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:21,048 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/u21i1vsf-media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 2024-12-31 21:07:21,201 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:22,220 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:22,781 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:23,206 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:23,769 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:07:24,113 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:07:24,113 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:07:24,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:24,781 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:25,223 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:26,231 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:26,249 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:27,207 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:28,208 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:28,783 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:28,783 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:07:29,021 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/a07y7rzg-events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:07:29,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:29,245 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: log_artifact 2024-12-31 21:07:29,245 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: log_artifact 2024-12-31 21:07:29,477 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:07:29,699 INFO wandb-upload_1:1740211 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpo_cu9z3a 2024-12-31 21:07:29,783 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:07:29,783 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2024-12-31 21:07:29,783 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media 2024-12-31 21:07:29,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval 2024-12-31 21:07:29,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table 2024-12-31 21:07:29,830 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: log_artifact 2024-12-31 21:07:29,842 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: partial_history 2024-12-31 21:07:30,102 INFO SenderThread:1740211 [sender.py:send_request_log_artifact():1525] logged artifact run-nkdxms1x-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDExNTA3Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2NjU0NQ==', 'latestArtifact': None}} 2024-12-31 21:07:30,104 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:07:30,104 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json with policy now 2024-12-31 21:07:30,105 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:07:30,105 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,106 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,107 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:07:30,107 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,108 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,108 DEBUG SenderThread:1740211 [sender.py:send():379] send: telemetry 2024-12-31 21:07:30,108 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,108 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,109 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,109 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,109 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,109 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,110 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,110 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,110 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,110 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,110 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,111 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,111 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,111 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,111 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,112 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,112 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,112 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,112 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,113 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,113 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,113 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:30,113 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: log_artifact 2024-12-31 21:07:30,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:30,277 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/ly18bboo-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2024-12-31 21:07:30,497 INFO wandb-upload_1:1740211 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpvrm_yjbf 2024-12-31 21:07:30,526 INFO wandb-upload_0:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/bhwrc5ru-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2024-12-31 21:07:30,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:30,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:30,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json 2024-12-31 21:07:30,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval 2024-12-31 21:07:30,850 INFO SenderThread:1740211 [sender.py:send_request_log_artifact():1525] logged artifact run-nkdxms1x-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDExNTA4NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2NjU0Nw==', 'latestArtifact': None}} 2024-12-31 21:07:30,851 DEBUG SenderThread:1740211 [sender.py:send():379] send: files 2024-12-31 21:07:30,851 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json with policy now 2024-12-31 21:07:30,851 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:07:30,851 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:30,851 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:31,032 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/089q6my2-media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json 2024-12-31 21:07:31,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:31,303 INFO wandb-upload_0:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/leshttye-media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json 2024-12-31 21:07:31,784 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:31,852 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:32,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:33,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:34,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:35,203 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:36,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:37,118 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:37,219 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:37,786 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/config.yaml 2024-12-31 21:07:38,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:39,097 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: stop_status 2024-12-31 21:07:39,097 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: stop_status 2024-12-31 21:07:39,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:40,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:41,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:42,204 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:43,069 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:43,110 DEBUG SenderThread:1740211 [sender.py:send():379] send: exit 2024-12-31 21:07:43,110 INFO SenderThread:1740211 [sender.py:send_exit():586] handling exit code: 0 2024-12-31 21:07:43,110 INFO SenderThread:1740211 [sender.py:send_exit():588] handling runtime: 259 2024-12-31 21:07:43,110 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:43,110 INFO SenderThread:1740211 [sender.py:send_exit():594] send defer 2024-12-31 21:07:43,110 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:43,110 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-31 21:07:43,111 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:43,111 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-31 21:07:43,111 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 1 2024-12-31 21:07:43,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:43,111 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-31 21:07:43,111 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:43,111 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-31 21:07:43,111 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 2 2024-12-31 21:07:43,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:43,111 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-31 21:07:43,111 INFO HandlerThread:1740211 [system_monitor.py:finish():203] Stopping system monitor 2024-12-31 21:07:43,111 DEBUG SystemMonitor:1740211 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-31 21:07:43,112 DEBUG SystemMonitor:1740211 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-31 21:07:43,112 INFO HandlerThread:1740211 [interfaces.py:finish():200] Joined cpu monitor 2024-12-31 21:07:43,112 INFO HandlerThread:1740211 [interfaces.py:finish():200] Joined disk monitor 2024-12-31 21:07:43,163 INFO HandlerThread:1740211 [interfaces.py:finish():200] Joined gpu monitor 2024-12-31 21:07:43,164 INFO HandlerThread:1740211 [interfaces.py:finish():200] Joined memory monitor 2024-12-31 21:07:43,164 INFO HandlerThread:1740211 [interfaces.py:finish():200] Joined network monitor 2024-12-31 21:07:43,164 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:43,164 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-31 21:07:43,164 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 3 2024-12-31 21:07:43,164 DEBUG SenderThread:1740211 [sender.py:send():379] send: stats 2024-12-31 21:07:43,164 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:43,165 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-31 21:07:43,165 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:43,165 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-31 21:07:43,165 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 4 2024-12-31 21:07:43,165 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:43,165 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-31 21:07:43,789 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:44,789 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:49,443 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:49,443 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:49,443 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: keepalive 2024-12-31 21:07:49,444 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:49,444 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-31 21:07:49,444 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 5 2024-12-31 21:07:49,444 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:49,444 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:49,444 DEBUG SenderThread:1740211 [sender.py:send():379] send: history 2024-12-31 21:07:49,444 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-31 21:07:49,445 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: summary_record 2024-12-31 21:07:49,446 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:49,446 DEBUG SenderThread:1740211 [sender.py:send():379] send: summary 2024-12-31 21:07:49,446 INFO SenderThread:1740211 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-31 21:07:49,446 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:49,446 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-31 21:07:49,446 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 6 2024-12-31 21:07:49,446 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:49,446 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-31 21:07:49,447 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:49,447 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-31 21:07:49,447 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 7 2024-12-31 21:07:49,447 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: status_report 2024-12-31 21:07:49,447 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:49,447 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-31 21:07:49,447 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:49,447 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-31 21:07:49,791 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:50,111 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:52,389 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 8 2024-12-31 21:07:52,389 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:52,389 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:52,389 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-31 21:07:52,389 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:52,389 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-31 21:07:52,389 INFO SenderThread:1740211 [job_builder.py:build():440] Attempting to build job artifact 2024-12-31 21:07:52,392 INFO SenderThread:1740211 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-31 21:07:52,393 INFO SenderThread:1740211 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-31 21:07:52,404 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 9 2024-12-31 21:07:52,404 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:52,404 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-31 21:07:52,404 DEBUG SenderThread:1740211 [sender.py:send():379] send: artifact 2024-12-31 21:07:52,753 INFO wandb-upload_1:1740211 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpaag6j_j3 2024-12-31 21:07:52,793 INFO Thread-12 :1740211 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:52,941 INFO wandb-upload_0:1740211 [upload_job.py:push():88] Uploaded file /tmp/tmp3rvw6g5e/wandb-job.json 2024-12-31 21:07:52,942 INFO wandb-upload_2:1740211 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp8_hforgk 2024-12-31 21:07:53,112 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:53,477 INFO SenderThread:1740211 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDExNTMxNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDExMjAxNQ==', 'versionIndex': 389}}} 2024-12-31 21:07:53,477 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:53,477 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-31 21:07:53,477 INFO SenderThread:1740211 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-31 21:07:53,793 INFO SenderThread:1740211 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/config.yaml config.yaml 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json wandb-summary.json 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/requirements.txt requirements.txt 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-metadata.json wandb-metadata.json 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log output.log 2024-12-31 21:07:53,794 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/diff.patch diff.patch 2024-12-31 21:07:53,796 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2024-12-31 21:07:53,798 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 media/videos/eval/4_eval_4_c43d65e2ea74e852f815.mp4 2024-12-31 21:07:53,798 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 media/videos/eval/1_eval_1_53a45cc58516bfa82806.mp4 2024-12-31 21:07:53,798 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 media/videos/eval/2_eval_2_fe1c0520eab448503096.mp4 2024-12-31 21:07:53,798 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/3_eval_3_a0e47918912079386856.mp4 media/videos/eval/3_eval_3_a0e47918912079386856.mp4 2024-12-31 21:07:53,801 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 media/videos/eval/0_eval_0_e739aa3a1064d7a115c6.mp4 2024-12-31 21:07:53,803 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json media/table/eval/total_success_rate_ci_6_9152843e0aab0648a7f1.table.json 2024-12-31 21:07:53,803 INFO SenderThread:1740211 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json 2024-12-31 21:07:53,803 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 10 2024-12-31 21:07:53,803 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:53,803 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:53,803 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-31 21:07:53,804 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:53,804 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-31 21:07:53,804 INFO SenderThread:1740211 [file_pusher.py:finish():169] shutting down file pusher 2024-12-31 21:07:53,970 INFO wandb-upload_0:1740211 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/wandb-summary.json 2024-12-31 21:07:54,008 INFO wandb-upload_3:1740211 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/requirements.txt 2024-12-31 21:07:54,029 INFO wandb-upload_1:1740211 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/config.yaml 2024-12-31 21:07:54,031 INFO wandb-upload_2:1740211 [upload_job.py:push():130] Uploaded file /tmp/tmpj0365mt3wandb/3jni8frz-events.out.tfevents.1735697004.leonmkim-ROG-Strix-G15CS-G15CS.1740046.0 2024-12-31 21:07:54,072 INFO wandb-upload_4:1740211 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/files/output.log 2024-12-31 21:07:54,113 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:54,113 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:54,272 INFO Thread-11 (_thread_body):1740211 [sender.py:transition_state():614] send defer: 11 2024-12-31 21:07:54,272 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:54,272 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-31 21:07:54,273 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:54,273 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-31 21:07:54,273 INFO SenderThread:1740211 [file_pusher.py:join():175] waiting for file pusher 2024-12-31 21:07:54,274 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 12 2024-12-31 21:07:54,274 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:54,274 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-31 21:07:54,274 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:54,275 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-31 21:07:54,275 INFO SenderThread:1740211 [file_stream.py:finish():601] file stream finish called 2024-12-31 21:07:54,347 INFO SenderThread:1740211 [file_stream.py:finish():605] file stream finish is done 2024-12-31 21:07:54,347 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 13 2024-12-31 21:07:54,347 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:54,347 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-31 21:07:54,347 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:54,347 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-31 21:07:54,347 INFO SenderThread:1740211 [sender.py:transition_state():614] send defer: 14 2024-12-31 21:07:54,348 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: defer 2024-12-31 21:07:54,348 INFO HandlerThread:1740211 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-31 21:07:54,348 DEBUG SenderThread:1740211 [sender.py:send():379] send: final 2024-12-31 21:07:54,348 DEBUG SenderThread:1740211 [sender.py:send():379] send: footer 2024-12-31 21:07:54,348 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: defer 2024-12-31 21:07:54,348 INFO SenderThread:1740211 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-31 21:07:54,348 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:54,349 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:54,349 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: server_info 2024-12-31 21:07:54,349 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-31 21:07:54,349 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: server_info 2024-12-31 21:07:54,349 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: get_summary 2024-12-31 21:07:54,351 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-31 21:07:54,351 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-31 21:07:54,401 DEBUG SenderThread:1740211 [sender.py:send_request():406] send_request: poll_exit 2024-12-31 21:07:54,401 INFO MainThread:1740211 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2024-12-31 21:07:54,402 INFO MainThread:1740211 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2024-12-31 21:07:54,402 INFO MainThread:1740211 [wandb_run.py:_footer_sync_info():3975] logging synced files 2024-12-31 21:07:54,402 DEBUG HandlerThread:1740211 [handler.py:handle_request():158] handle_request: shutdown 2024-12-31 21:07:54,402 INFO HandlerThread:1740211 [handler.py:finish():882] shutting down handler 2024-12-31 21:07:55,352 INFO WriterThread:1740211 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/nkdxms1x/wandb/run-20241231_210323-nkdxms1x/run-nkdxms1x.wandb 2024-12-31 21:07:55,401 INFO SenderThread:1740211 [sender.py:finish():1615] shutting down sender 2024-12-31 21:07:55,401 INFO SenderThread:1740211 [file_pusher.py:finish():169] shutting down file pusher 2024-12-31 21:07:55,401 INFO SenderThread:1740211 [file_pusher.py:join():175] waiting for file pusher