2025-01-01 10:57:41,755 INFO StreamThr :1788056 [internal.py:wandb_internal():85] W&B internal server running at pid: 1788056, started at: 2025-01-01 10:57:41.755427 2025-01-01 10:57:41,756 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status 2025-01-01 10:57:41,768 INFO WriterThread:1788056 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/run-6necfy24.wandb 2025-01-01 10:57:41,769 DEBUG SenderThread:1788056 [sender.py:send():379] send: header 2025-01-01 10:57:41,778 DEBUG SenderThread:1788056 [sender.py:send():379] send: run 2025-01-01 10:57:41,983 INFO SenderThread:1788056 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files 2025-01-01 10:57:41,983 INFO SenderThread:1788056 [sender.py:_start_run_threads():1188] run started: 6necfy24 with start time 1735747061.75616 2025-01-01 10:57:41,999 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: check_version 2025-01-01 10:57:41,999 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: check_version 2025-01-01 10:57:42,043 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: run_start 2025-01-01 10:57:42,048 DEBUG HandlerThread:1788056 [system_info.py:__init__():26] System info init 2025-01-01 10:57:42,048 DEBUG HandlerThread:1788056 [system_info.py:__init__():41] System info init done 2025-01-01 10:57:42,048 INFO HandlerThread:1788056 [system_monitor.py:start():194] Starting system monitor 2025-01-01 10:57:42,048 INFO SystemMonitor:1788056 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-01 10:57:42,049 INFO HandlerThread:1788056 [system_monitor.py:probe():214] Collecting system info 2025-01-01 10:57:42,049 INFO SystemMonitor:1788056 [interfaces.py:start():188] Started cpu monitoring 2025-01-01 10:57:42,050 INFO SystemMonitor:1788056 [interfaces.py:start():188] Started disk monitoring 2025-01-01 10:57:42,050 INFO SystemMonitor:1788056 [interfaces.py:start():188] Started gpu monitoring 2025-01-01 10:57:42,051 INFO SystemMonitor:1788056 [interfaces.py:start():188] Started memory monitoring 2025-01-01 10:57:42,051 INFO SystemMonitor:1788056 [interfaces.py:start():188] Started network monitoring 2025-01-01 10:57:42,060 DEBUG HandlerThread:1788056 [system_info.py:probe():152] Probing system 2025-01-01 10:57:42,062 DEBUG HandlerThread:1788056 [system_info.py:_probe_git():137] Probing git 2025-01-01 10:57:42,066 DEBUG HandlerThread:1788056 [system_info.py:_probe_git():145] Probing git done 2025-01-01 10:57:42,066 DEBUG HandlerThread:1788056 [system_info.py:probe():200] Probing system done 2025-01-01 10:57:42,066 DEBUG HandlerThread:1788056 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T15:57:42.060972', 'startedAt': '2025-01-01T15:57:41.743321', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.4958749999996, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}, {'current': 4500.008, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 657.4940643310547}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-01 10:57:42,066 INFO HandlerThread:1788056 [system_monitor.py:probe():224] Finished collecting system info 2025-01-01 10:57:42,066 INFO HandlerThread:1788056 [system_monitor.py:probe():227] Publishing system info 2025-01-01 10:57:42,067 DEBUG HandlerThread:1788056 [system_info.py:_save_code():44] Saving code 2025-01-01 10:57:42,068 DEBUG HandlerThread:1788056 [system_info.py:_save_code():67] Saving code done 2025-01-01 10:57:42,068 DEBUG HandlerThread:1788056 [system_info.py:_save_patches():84] Saving git patches 2025-01-01 10:57:42,130 DEBUG HandlerThread:1788056 [system_info.py:_save_patches():126] Saving git patches done 2025-01-01 10:57:42,131 INFO HandlerThread:1788056 [system_monitor.py:probe():229] Finished publishing system info 2025-01-01 10:57:42,134 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 10:57:42,134 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-01 10:57:42,134 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-01 10:57:42,134 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-01 10:57:42,374 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: python_packages 2025-01-01 10:57:42,374 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: python_packages 2025-01-01 10:57:42,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:57:42,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:57:42,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:42,379 INFO HandlerThread:1788056 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/tb" save: true } control { relay_id: "e70ba2374c284bda9de162ecdf68005d" } _info { stream_id: "6necfy24" } 2025-01-01 10:57:42,379 DEBUG HandlerThread:1788056 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-01 10:57:42,390 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/j1t6nnty-code/FISH/eval_robot.py 2025-01-01 10:57:42,392 INFO wandb-upload_0:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/kmz6eb0n-wandb-metadata.json 2025-01-01 10:57:42,400 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/vzu4x605-diff.patch 2025-01-01 10:57:42,506 DEBUG SenderThread:1788056 [sender.py:send():379] send: telemetry 2025-01-01 10:57:42,506 DEBUG SenderThread:1788056 [sender.py:send():379] send: tbrecord 2025-01-01 10:57:42,506 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 10:57:42,506 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 with policy live 2025-01-01 10:57:42,689 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/p0ywqfw0-events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:57:42,985 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/code/FISH/eval_robot.py 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/requirements.txt 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-metadata.json 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/diff.patch 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/code/FISH 2025-01-01 10:57:42,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/code 2025-01-01 10:57:43,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:44,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:44,984 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:45,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:46,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:47,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:57:47,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:48,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:49,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:49,447 DEBUG SenderThread:1788056 [sender.py:send():379] send: config 2025-01-01 10:57:50,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:50,985 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:51,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:52,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:57:52,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:52,986 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:53,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:54,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:54,987 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:55,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:56,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:56,987 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:57,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:57:57,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:57:57,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:57,464 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:57:58,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:57:58,988 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:57:59,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:00,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:01,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:02,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:03,253 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:03,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:04,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:04,990 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:05,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:06,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:06,990 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:07,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:08,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:08,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:08,991 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:09,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:10,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:11,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:12,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:58:12,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:58:12,417 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:13,387 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:13,445 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:13,992 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/config.yaml 2025-01-01 10:58:14,388 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:15,388 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:16,388 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:17,388 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:18,388 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:19,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:19,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:20,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:21,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:22,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:23,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:24,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:24,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:25,389 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:26,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:27,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:58:27,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:58:27,417 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:28,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:29,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:29,464 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:30,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:31,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:32,391 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:33,392 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:33,931 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 10:58:33,931 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 with policy now 2025-01-01 10:58:33,931 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 10:58:33,932 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 10:58:33,934 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 10:58:33,935 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 10:58:33,997 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 2025-01-01 10:58:33,998 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 10:58:33,998 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval 2025-01-01 10:58:33,998 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos 2025-01-01 10:58:33,998 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media 2025-01-01 10:58:34,188 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/eyx99nu8-media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 2025-01-01 10:58:34,392 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:34,449 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/0abzwge7-media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 2025-01-01 10:58:34,942 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:34,998 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:35,391 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:36,393 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:37,394 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:38,393 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:39,394 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:40,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:40,493 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:41,000 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:41,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:42,051 DEBUG SystemMonitor:1788056 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-01 10:58:42,052 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 10:58:42,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:58:42,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:58:42,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:43,000 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:43,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:44,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:45,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:46,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:46,491 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:47,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:48,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:49,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:50,003 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:58:50,237 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/djkmibn0-events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:58:50,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:51,003 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:51,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:51,817 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:52,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:53,004 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:53,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:54,396 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:55,005 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:58:55,397 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:56,397 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:57,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:58:57,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:58:57,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:57,527 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:58:58,398 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:58:59,398 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:00,398 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:01,398 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:02,398 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:03,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:03,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:04,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:05,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:06,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:07,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:08,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:08,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:09,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:10,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:11,402 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:12,053 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 10:59:12,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:59:12,375 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:59:12,417 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:13,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:13,440 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:14,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:15,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:16,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:17,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:18,403 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:19,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:19,404 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:20,404 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:21,408 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:21,465 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 10:59:21,465 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 10:59:21,465 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 with policy now 2025-01-01 10:59:21,473 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 10:59:21,473 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 10:59:21,475 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 10:59:21,718 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/rhw6z3fz-media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 2025-01-01 10:59:22,013 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 10:59:22,013 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 2025-01-01 10:59:22,013 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval 2025-01-01 10:59:22,301 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/thheyvvn-media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 2025-01-01 10:59:22,409 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:23,013 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:59:23,405 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:24,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:24,411 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:25,415 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:26,417 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:27,015 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:59:27,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:59:27,378 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:59:27,425 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:28,410 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:29,015 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:59:29,410 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:29,982 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:30,412 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:31,415 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:32,410 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:33,410 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:34,410 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:35,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:35,411 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:36,412 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:37,411 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:38,411 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:39,411 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:40,412 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:41,380 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:41,412 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:42,053 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 10:59:42,377 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:59:42,377 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:59:42,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:43,413 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:44,413 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:45,413 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:46,414 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:46,469 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:47,414 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:48,414 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:49,415 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:50,416 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:51,415 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:52,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:52,414 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:52,812 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 10:59:52,812 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 10:59:52,812 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 10:59:53,024 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:59:53,024 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 10:59:53,267 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/2klaftrg-events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 10:59:53,415 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:54,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:55,025 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:59:55,419 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:56,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:57,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 10:59:57,376 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 10:59:57,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:57,481 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 10:59:58,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 10:59:59,026 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 10:59:59,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:00,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:01,027 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:01,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:02,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:03,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:03,420 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:04,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:05,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:06,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:07,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:08,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:09,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:09,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:10,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:11,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:12,053 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:00:12,376 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:00:12,377 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:00:12,421 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:13,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:14,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:14,459 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:15,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:16,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:17,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:18,422 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:19,423 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:19,917 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:20,423 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:21,034 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:21,423 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:22,423 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:23,437 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:24,435 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:24,745 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 11:00:24,746 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 with policy now 2025-01-01 11:00:24,750 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 11:00:24,750 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:00:24,750 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:00:24,750 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:00:24,991 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/yj2aawlf-media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 2025-01-01 11:00:25,035 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:25,036 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:00:25,036 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 2025-01-01 11:00:25,036 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval 2025-01-01 11:00:25,315 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/gq70ldhr-media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 2025-01-01 11:00:25,436 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:25,769 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:26,436 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:27,036 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:27,379 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:00:27,379 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:00:27,438 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:28,437 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:29,440 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:30,438 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:30,825 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:31,038 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:31,441 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:32,441 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:33,441 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:34,441 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:35,039 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:35,440 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:36,213 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:36,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:37,040 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:37,447 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:38,447 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:38,647 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:00:38,647 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:00:38,647 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:00:39,040 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:00:39,040 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:00:39,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:40,453 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:41,041 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:41,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:41,613 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:42,054 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:00:42,397 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:00:42,398 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:00:42,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:43,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:44,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:45,042 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:00:45,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:46,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:46,850 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:47,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:48,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:49,449 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:50,450 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:51,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:52,385 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:52,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:53,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:54,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:55,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:56,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:57,378 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:00:57,378 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:00:57,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:57,503 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:00:58,451 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:00:59,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:00,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:01,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:02,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:03,385 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:03,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:04,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:05,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:06,452 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:07,453 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:08,385 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:08,453 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:09,453 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:10,453 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:11,457 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:11,984 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 11:01:11,985 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 with policy now 2025-01-01 11:01:11,988 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 11:01:11,989 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:01:11,989 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:01:11,990 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:01:12,051 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:01:12,051 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 2025-01-01 11:01:12,052 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval 2025-01-01 11:01:12,054 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:01:12,241 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/9u3y1704-media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 2025-01-01 11:01:12,384 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:01:12,384 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:01:12,459 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:12,500 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/3nl4yisw-media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 2025-01-01 11:01:13,052 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:13,464 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:13,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:14,475 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:15,480 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:16,469 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:17,479 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:18,468 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:19,054 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:19,375 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:19,463 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:20,466 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:21,054 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:21,464 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:22,473 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:23,468 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:24,391 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:24,466 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:25,467 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:26,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:27,386 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:01:27,386 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:01:27,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:28,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:29,057 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:01:29,293 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/mlsq6ykk-events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:01:29,479 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:29,708 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:01:29,709 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:01:29,709 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:29,709 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:01:30,057 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:01:30,479 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:31,058 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:31,473 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:32,474 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:33,058 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:33,469 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:34,469 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:34,929 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:35,059 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:35,469 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:36,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:37,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:38,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:39,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:40,390 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:40,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:41,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:42,055 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:01:42,381 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:01:42,382 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:01:42,470 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:43,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:44,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:45,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:45,525 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:46,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:47,471 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:48,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:49,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:50,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:51,391 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:51,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:52,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:53,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:54,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:55,066 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:01:55,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:56,472 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:56,910 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:01:57,382 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:01:57,382 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:01:57,473 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:58,473 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:01:59,473 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:00,477 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:01,068 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:01,203 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 11:02:01,203 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 with policy now 2025-01-01 11:02:01,205 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 11:02:01,206 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:02:01,206 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:01,206 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:01,458 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/mby126yl-media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 2025-01-01 11:02:01,474 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:02,069 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:02,069 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 2025-01-01 11:02:02,069 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval 2025-01-01 11:02:02,238 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:02,369 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/d2f2eaxf-media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 2025-01-01 11:02:02,486 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:03,069 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:03,483 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:04,484 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:05,490 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:06,485 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:07,071 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:07,395 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:07,494 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:08,494 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:09,497 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:10,499 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:11,072 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:11,498 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:12,055 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:02:12,397 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: stop_status 2025-01-01 11:02:12,398 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: stop_status 2025-01-01 11:02:12,496 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:12,499 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:13,496 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:14,498 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:15,499 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:16,503 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:17,513 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:17,654 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-01 11:02:17,654 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: log_artifact 2025-01-01 11:02:17,832 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 11:02:18,074 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:02:18,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json 2025-01-01 11:02:18,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media 2025-01-01 11:02:18,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table 2025-01-01 11:02:18,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval 2025-01-01 11:02:18,124 INFO wandb-upload_1:1788056 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpd7jhhpw5 2025-01-01 11:02:18,236 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-01 11:02:18,244 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: partial_history 2025-01-01 11:02:18,286 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/qpqsfr8z-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json 2025-01-01 11:02:18,497 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:18,598 INFO SenderThread:1788056 [sender.py:send_request_log_artifact():1525] logged artifact run-6necfy24-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY3MDEzNQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTIzOTU4MQ==', 'latestArtifact': None}} 2025-01-01 11:02:18,598 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 11:02:18,598 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json with policy now 2025-01-01 11:02:18,598 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:18,599 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:02:18,599 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,599 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,600 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:02:18,600 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,603 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,604 DEBUG SenderThread:1788056 [sender.py:send():379] send: telemetry 2025-01-01 11:02:18,604 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,604 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,604 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,604 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,605 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,605 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,605 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,605 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,606 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,606 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,606 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,606 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,607 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,607 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,607 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,607 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,608 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,608 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,608 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,608 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,609 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:18,609 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:18,609 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: log_artifact 2025-01-01 11:02:18,846 INFO wandb-upload_0:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/pc4x4hed-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json 2025-01-01 11:02:18,959 INFO wandb-upload_2:1788056 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpktagczqq 2025-01-01 11:02:19,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:02:19,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:19,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:19,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json 2025-01-01 11:02:19,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval 2025-01-01 11:02:19,339 INFO SenderThread:1788056 [sender.py:send_request_log_artifact():1525] logged artifact run-6necfy24-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY3MDE0NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTIzOTU4NA==', 'latestArtifact': None}} 2025-01-01 11:02:19,340 DEBUG SenderThread:1788056 [sender.py:send():379] send: files 2025-01-01 11:02:19,340 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json with policy now 2025-01-01 11:02:19,340 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:02:19,340 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:19,341 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:19,497 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:19,518 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/712cyue4-media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json 2025-01-01 11:02:19,774 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/1xs3cf6p-media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json 2025-01-01 11:02:20,075 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:20,497 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:20,609 DEBUG SenderThread:1788056 [sender.py:send():379] send: exit 2025-01-01 11:02:20,609 INFO SenderThread:1788056 [sender.py:send_exit():586] handling exit code: 0 2025-01-01 11:02:20,610 INFO SenderThread:1788056 [sender.py:send_exit():588] handling runtime: 278 2025-01-01 11:02:20,610 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:20,610 INFO SenderThread:1788056 [sender.py:send_exit():594] send defer 2025-01-01 11:02:20,610 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:20,610 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-01 11:02:20,610 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:20,610 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-01 11:02:20,610 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 1 2025-01-01 11:02:20,611 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:20,611 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-01 11:02:20,611 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:20,611 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-01 11:02:20,611 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 2 2025-01-01 11:02:20,611 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:20,611 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-01 11:02:20,611 INFO HandlerThread:1788056 [system_monitor.py:finish():203] Stopping system monitor 2025-01-01 11:02:20,611 INFO HandlerThread:1788056 [interfaces.py:finish():200] Joined cpu monitor 2025-01-01 11:02:20,612 DEBUG SystemMonitor:1788056 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-01 11:02:20,612 DEBUG SystemMonitor:1788056 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-01 11:02:20,612 INFO HandlerThread:1788056 [interfaces.py:finish():200] Joined disk monitor 2025-01-01 11:02:20,663 INFO HandlerThread:1788056 [interfaces.py:finish():200] Joined gpu monitor 2025-01-01 11:02:20,664 INFO HandlerThread:1788056 [interfaces.py:finish():200] Joined memory monitor 2025-01-01 11:02:20,664 INFO HandlerThread:1788056 [interfaces.py:finish():200] Joined network monitor 2025-01-01 11:02:20,664 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:20,664 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-01 11:02:20,664 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 3 2025-01-01 11:02:20,664 DEBUG SenderThread:1788056 [sender.py:send():379] send: stats 2025-01-01 11:02:20,664 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:20,665 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-01 11:02:20,665 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:20,665 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-01 11:02:20,665 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 4 2025-01-01 11:02:20,665 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:20,665 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-01 11:02:21,076 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:21,076 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:23,077 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:26,771 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-01 11:02:26,771 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:26,771 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: keepalive 2025-01-01 11:02:26,772 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:26,772 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-01 11:02:26,772 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 5 2025-01-01 11:02:26,772 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: poll_exit 2025-01-01 11:02:26,773 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:26,773 DEBUG SenderThread:1788056 [sender.py:send():379] send: history 2025-01-01 11:02:26,773 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-01 11:02:26,773 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: summary_record 2025-01-01 11:02:26,774 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:26,774 DEBUG SenderThread:1788056 [sender.py:send():379] send: summary 2025-01-01 11:02:26,774 INFO SenderThread:1788056 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-01 11:02:26,774 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:26,775 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-01 11:02:26,775 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 6 2025-01-01 11:02:26,775 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:26,775 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-01 11:02:26,775 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:26,775 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-01 11:02:26,778 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: status_report 2025-01-01 11:02:26,878 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 7 2025-01-01 11:02:26,878 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:26,878 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-01 11:02:26,879 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:26,879 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-01 11:02:27,078 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/config.yaml 2025-01-01 11:02:27,078 INFO Thread-12 :1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:27,610 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-01 11:02:30,743 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 8 2025-01-01 11:02:30,743 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: poll_exit 2025-01-01 11:02:30,744 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:30,744 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-01 11:02:30,744 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:30,744 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-01 11:02:30,744 INFO SenderThread:1788056 [job_builder.py:build():440] Attempting to build job artifact 2025-01-01 11:02:30,746 INFO SenderThread:1788056 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-01 11:02:30,748 INFO SenderThread:1788056 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-01 11:02:30,757 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 9 2025-01-01 11:02:30,757 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:30,758 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-01 11:02:30,758 DEBUG SenderThread:1788056 [sender.py:send():379] send: artifact 2025-01-01 11:02:30,936 INFO SenderThread:1788056 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'versionIndex': 404}}} 2025-01-01 11:02:30,937 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:30,937 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-01 11:02:30,937 INFO SenderThread:1788056 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-01 11:02:31,079 INFO SenderThread:1788056 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:31,079 INFO SenderThread:1788056 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/config.yaml config.yaml 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json wandb-summary.json 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/requirements.txt requirements.txt 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-metadata.json wandb-metadata.json 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log output.log 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/diff.patch diff.patch 2025-01-01 11:02:31,080 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-01 11:02:31,084 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 media/videos/eval/4_eval_4_6071852b701bfb891669.mp4 2025-01-01 11:02:31,084 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 media/videos/eval/0_eval_0_259d97e9bef5a9deb167.mp4 2025-01-01 11:02:31,084 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 media/videos/eval/3_eval_3_1c141374c1432359d5cd.mp4 2025-01-01 11:02:31,084 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 media/videos/eval/1_eval_1_09700a0ebedadd7f6657.mp4 2025-01-01 11:02:31,084 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 media/videos/eval/2_eval_2_3ac987b25093947b056a.mp4 2025-01-01 11:02:31,086 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json media/table/eval/total_success_rate_ci_6_b9cc57e1cd93555a8ecd.table.json 2025-01-01 11:02:31,086 INFO SenderThread:1788056 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json 2025-01-01 11:02:31,086 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 10 2025-01-01 11:02:31,087 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:31,089 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-01 11:02:31,089 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:31,089 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-01 11:02:31,089 INFO SenderThread:1788056 [file_pusher.py:finish():169] shutting down file pusher 2025-01-01 11:02:31,263 INFO wandb-upload_1:1788056 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/config.yaml 2025-01-01 11:02:31,283 INFO wandb-upload_0:1788056 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/wandb-summary.json 2025-01-01 11:02:31,311 INFO wandb-upload_4:1788056 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/output.log 2025-01-01 11:02:31,321 INFO wandb-upload_2:1788056 [upload_job.py:push():130] Uploaded file /tmp/tmpt9_hvm8owandb/831ywcq5-events.out.tfevents.1735747062.leonmkim-ROG-Strix-G15CS-G15CS.1787880.0 2025-01-01 11:02:31,338 INFO wandb-upload_3:1788056 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/files/requirements.txt 2025-01-01 11:02:31,538 INFO Thread-11 (_thread_body):1788056 [sender.py:transition_state():614] send defer: 11 2025-01-01 11:02:31,538 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:31,538 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-01 11:02:31,539 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:31,539 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-01 11:02:31,539 INFO SenderThread:1788056 [file_pusher.py:join():175] waiting for file pusher 2025-01-01 11:02:31,540 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 12 2025-01-01 11:02:31,540 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:31,540 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-01 11:02:31,540 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:31,541 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-01 11:02:31,541 INFO SenderThread:1788056 [file_stream.py:finish():601] file stream finish called 2025-01-01 11:02:31,601 INFO SenderThread:1788056 [file_stream.py:finish():605] file stream finish is done 2025-01-01 11:02:31,602 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 13 2025-01-01 11:02:31,602 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:31,602 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-01 11:02:31,602 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:31,602 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-01 11:02:31,602 INFO SenderThread:1788056 [sender.py:transition_state():614] send defer: 14 2025-01-01 11:02:31,602 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: defer 2025-01-01 11:02:31,602 INFO HandlerThread:1788056 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-01 11:02:31,603 DEBUG SenderThread:1788056 [sender.py:send():379] send: final 2025-01-01 11:02:31,603 DEBUG SenderThread:1788056 [sender.py:send():379] send: footer 2025-01-01 11:02:31,603 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: defer 2025-01-01 11:02:31,603 INFO SenderThread:1788056 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-01 11:02:31,603 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-01 11:02:31,604 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: server_info 2025-01-01 11:02:31,604 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: poll_exit 2025-01-01 11:02:31,604 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-01 11:02:31,604 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: server_info 2025-01-01 11:02:31,606 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: get_summary 2025-01-01 11:02:31,606 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-01 11:02:31,607 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-01 11:02:31,658 DEBUG SenderThread:1788056 [sender.py:send_request():406] send_request: poll_exit 2025-01-01 11:02:31,658 INFO MainThread:1788056 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-01 11:02:31,659 INFO MainThread:1788056 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-01 11:02:31,659 INFO MainThread:1788056 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-01 11:02:31,659 DEBUG HandlerThread:1788056 [handler.py:handle_request():158] handle_request: shutdown 2025-01-01 11:02:31,659 INFO HandlerThread:1788056 [handler.py:finish():882] shutting down handler 2025-01-01 11:02:32,605 INFO WriterThread:1788056 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/6necfy24/wandb/run-20250101_105741-6necfy24/run-6necfy24.wandb 2025-01-01 11:02:32,658 INFO SenderThread:1788056 [sender.py:finish():1615] shutting down sender 2025-01-01 11:02:32,658 INFO SenderThread:1788056 [file_pusher.py:finish():169] shutting down file pusher 2025-01-01 11:02:32,658 INFO SenderThread:1788056 [file_pusher.py:join():175] waiting for file pusher