2025-01-06 17:07:42,087 INFO StreamThr :2085739 [internal.py:wandb_internal():85] W&B internal server running at pid: 2085739, started at: 2025-01-06 17:07:42.087026 2025-01-06 17:07:42,088 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status 2025-01-06 17:07:42,099 INFO WriterThread:2085739 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/run-8ycpgp3n.wandb 2025-01-06 17:07:42,099 DEBUG SenderThread:2085739 [sender.py:send():379] send: header 2025-01-06 17:07:42,108 DEBUG SenderThread:2085739 [sender.py:send():379] send: run 2025-01-06 17:07:42,367 INFO SenderThread:2085739 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files 2025-01-06 17:07:42,367 INFO SenderThread:2085739 [sender.py:_start_run_threads():1188] run started: 8ycpgp3n with start time 1736201262.087823 2025-01-06 17:07:42,383 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: check_version 2025-01-06 17:07:42,384 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: check_version 2025-01-06 17:07:42,456 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: run_start 2025-01-06 17:07:42,461 DEBUG HandlerThread:2085739 [system_info.py:__init__():26] System info init 2025-01-06 17:07:42,461 DEBUG HandlerThread:2085739 [system_info.py:__init__():41] System info init done 2025-01-06 17:07:42,461 INFO HandlerThread:2085739 [system_monitor.py:start():194] Starting system monitor 2025-01-06 17:07:42,461 INFO SystemMonitor:2085739 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-06 17:07:42,461 INFO HandlerThread:2085739 [system_monitor.py:probe():214] Collecting system info 2025-01-06 17:07:42,462 INFO SystemMonitor:2085739 [interfaces.py:start():188] Started cpu monitoring 2025-01-06 17:07:42,463 INFO SystemMonitor:2085739 [interfaces.py:start():188] Started disk monitoring 2025-01-06 17:07:42,463 INFO SystemMonitor:2085739 [interfaces.py:start():188] Started gpu monitoring 2025-01-06 17:07:42,464 INFO SystemMonitor:2085739 [interfaces.py:start():188] Started memory monitoring 2025-01-06 17:07:42,464 INFO SystemMonitor:2085739 [interfaces.py:start():188] Started network monitoring 2025-01-06 17:07:42,474 DEBUG HandlerThread:2085739 [system_info.py:probe():152] Probing system 2025-01-06 17:07:42,476 DEBUG HandlerThread:2085739 [system_info.py:_probe_git():137] Probing git 2025-01-06 17:07:42,481 DEBUG HandlerThread:2085739 [system_info.py:_probe_git():145] Probing git done 2025-01-06 17:07:42,481 DEBUG HandlerThread:2085739 [system_info.py:probe():200] Probing system done 2025-01-06 17:07:42,481 DEBUG HandlerThread:2085739 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-06T22:07:42.474816', 'startedAt': '2025-01-06T22:07:42.079437', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3937.0075, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.117, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.969, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 314.2787742614746}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-06 17:07:42,481 INFO HandlerThread:2085739 [system_monitor.py:probe():224] Finished collecting system info 2025-01-06 17:07:42,481 INFO HandlerThread:2085739 [system_monitor.py:probe():227] Publishing system info 2025-01-06 17:07:42,481 DEBUG HandlerThread:2085739 [system_info.py:_save_code():44] Saving code 2025-01-06 17:07:42,482 DEBUG HandlerThread:2085739 [system_info.py:_save_code():67] Saving code done 2025-01-06 17:07:42,482 DEBUG HandlerThread:2085739 [system_info.py:_save_patches():84] Saving git patches 2025-01-06 17:07:42,544 DEBUG HandlerThread:2085739 [system_info.py:_save_patches():126] Saving git patches done 2025-01-06 17:07:42,544 INFO HandlerThread:2085739 [system_monitor.py:probe():229] Finished publishing system info 2025-01-06 17:07:42,547 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:07:42,547 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-06 17:07:42,548 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-06 17:07:42,548 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-06 17:07:42,789 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: python_packages 2025-01-06 17:07:42,789 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: python_packages 2025-01-06 17:07:42,789 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:07:42,789 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:07:42,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:42,792 INFO HandlerThread:2085739 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/tb" save: true } control { relay_id: "f007897853a543cc8f6f92970a65dbe3" } _info { stream_id: "8ycpgp3n" } 2025-01-06 17:07:42,792 DEBUG HandlerThread:2085739 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-06 17:07:42,799 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/hgz8a79o-wandb-metadata.json 2025-01-06 17:07:42,799 INFO wandb-upload_2:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/m8dd2rbx-diff.patch 2025-01-06 17:07:42,848 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/mplv54em-code/FISH/eval_robot.py 2025-01-06 17:07:42,877 DEBUG SenderThread:2085739 [sender.py:send():379] send: telemetry 2025-01-06 17:07:42,877 DEBUG SenderThread:2085739 [sender.py:send():379] send: tbrecord 2025-01-06 17:07:42,878 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:07:42,878 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 with policy live 2025-01-06 17:07:43,070 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/bikec69p-events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:07:43,369 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/diff.patch 2025-01-06 17:07:43,369 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/requirements.txt 2025-01-06 17:07:43,369 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:07:43,369 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/code/FISH/eval_robot.py 2025-01-06 17:07:43,370 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-metadata.json 2025-01-06 17:07:43,370 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:43,370 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/code/FISH 2025-01-06 17:07:43,370 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/code 2025-01-06 17:07:43,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:44,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:45,368 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:45,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:46,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:47,332 DEBUG SenderThread:2085739 [sender.py:send():379] send: config 2025-01-06 17:07:47,332 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:07:47,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:48,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:49,369 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:49,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:50,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:51,370 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:51,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:52,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:07:52,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:53,371 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:53,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:54,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:55,371 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:55,813 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:56,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:57,372 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:07:57,789 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:07:57,789 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:07:57,833 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:57,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:07:58,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:07:59,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:00,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:01,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:02,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:03,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:03,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:04,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:05,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:06,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:07,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:08,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:08,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:09,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:10,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:11,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:12,789 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:08:12,789 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:08:12,833 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:13,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:13,939 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:14,376 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/config.yaml 2025-01-06 17:08:14,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:15,377 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:15,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:16,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:17,377 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:17,794 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:18,794 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:19,326 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:19,381 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:19,797 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:20,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:21,797 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:22,797 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:23,797 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:24,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:24,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:25,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:26,807 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:27,789 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:08:27,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:08:27,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:28,807 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:29,807 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:29,851 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:30,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:31,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:32,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:33,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:34,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:35,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:35,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:36,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:37,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:38,808 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:39,809 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:40,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:40,809 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:41,809 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:42,465 DEBUG SystemMonitor:2085739 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-06 17:08:42,465 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:08:42,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:08:42,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:08:42,833 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:43,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:44,816 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:45,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:45,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:46,004 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:46,049 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:08:46,050 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:08:46,050 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 with policy now 2025-01-06 17:08:46,050 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:08:46,050 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:08:46,051 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:08:46,310 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/oiocex7m-media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 2025-01-06 17:08:46,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 2025-01-06 17:08:46,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:08:46,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media 2025-01-06 17:08:46,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval 2025-01-06 17:08:46,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos 2025-01-06 17:08:46,665 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/igx7hdy1-media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 2025-01-06 17:08:46,816 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:47,388 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:47,816 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:48,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:49,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:50,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:51,390 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:51,669 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:51,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:52,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:53,390 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:53,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:54,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:55,391 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:08:55,646 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/s92ufxho-events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:08:55,817 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:56,677 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:08:56,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:57,392 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:57,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:08:57,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:08:57,833 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:58,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:08:59,393 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:08:59,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:00,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:01,394 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:01,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:02,589 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:02,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:03,819 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:04,819 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:05,395 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:05,819 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:06,819 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:07,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:07,819 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:08,820 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:09,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:10,820 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:11,820 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:12,466 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:09:12,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:09:12,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:09:12,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:12,935 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:13,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:14,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:15,398 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:15,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:16,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:17,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:18,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:18,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:19,821 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:20,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:21,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:22,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:23,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:24,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:24,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:25,823 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:26,627 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:09:26,627 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 with policy now 2025-01-06 17:09:26,632 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:09:26,633 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:09:26,633 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:09:26,634 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:09:26,822 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:26,894 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/x1fr96t6-media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 2025-01-06 17:09:27,402 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:27,403 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:09:27,403 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 2025-01-06 17:09:27,403 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval 2025-01-06 17:09:27,688 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/yjfmt6dg-media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 2025-01-06 17:09:27,799 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:09:27,799 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:09:27,841 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:28,834 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:29,403 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:29,830 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:29,930 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:30,831 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:31,404 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:31,833 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:32,834 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:33,834 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:34,834 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:35,405 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:35,834 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:35,885 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:36,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:37,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:38,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:39,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:40,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:41,795 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:41,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:42,408 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:09:42,466 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:09:42,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:09:42,791 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:09:42,835 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:43,217 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:09:43,218 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:09:43,218 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:09:43,408 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:43,409 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:09:43,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:44,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:45,409 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:45,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:46,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:47,410 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:09:47,428 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:47,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:48,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:49,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:50,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:51,836 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:52,795 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:52,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:53,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:54,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:55,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:56,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:57,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:09:57,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:09:57,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:58,220 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:09:58,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:09:59,837 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:00,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:01,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:02,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:03,795 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:03,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:04,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:05,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:06,838 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:07,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:08,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:09,796 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:09,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:10,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:11,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:12,467 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:10:12,790 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:10:12,790 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:10:12,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:13,842 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:14,633 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:10:14,633 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 with policy now 2025-01-06 17:10:14,636 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:10:14,637 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:10:14,637 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:10:14,637 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:10:14,840 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:14,928 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/6tlyz576-media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 2025-01-06 17:10:15,419 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:15,420 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:10:15,420 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 2025-01-06 17:10:15,420 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval 2025-01-06 17:10:15,659 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:15,714 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/e2j46bsj-media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 2025-01-06 17:10:15,841 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:16,842 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:17,420 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:17,847 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:18,847 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:19,421 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:19,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:20,847 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:20,849 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:21,851 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:22,855 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:23,422 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:23,855 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:24,851 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:25,851 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:26,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:26,852 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:27,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:10:27,793 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:10:27,854 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:28,853 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:29,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:30,859 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:31,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:31,917 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:32,428 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:10:32,678 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/g35elcwn-events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:10:32,855 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:33,051 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:10:33,051 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:10:33,052 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:10:33,428 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:33,429 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:10:33,855 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:34,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:35,429 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:35,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:36,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:37,081 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:37,430 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:37,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:38,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:39,431 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:10:39,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:40,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:41,856 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:42,467 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:10:42,468 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:42,792 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:10:42,792 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:10:42,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:43,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:44,857 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:45,863 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:46,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:47,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:47,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:48,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:49,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:50,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:51,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:52,798 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:52,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:53,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:54,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:55,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:56,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:57,791 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:10:57,791 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:10:57,864 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:10:57,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:58,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:10:59,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:00,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:01,865 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:02,867 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:03,799 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:03,867 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:04,871 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:05,329 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:11:05,329 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 with policy now 2025-01-06 17:11:05,332 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:11:05,332 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:11:05,332 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:11:05,333 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:11:05,445 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:05,445 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:11:05,445 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 2025-01-06 17:11:05,445 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval 2025-01-06 17:11:05,580 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/eqwxx7lq-media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 2025-01-06 17:11:05,843 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/niwyj3l8-media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 2025-01-06 17:11:05,877 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:06,881 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:07,446 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:07,875 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:08,875 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:09,797 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:09,878 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:10,879 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:11,447 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:11,872 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:12,468 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:11:12,810 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:11:12,810 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:11:12,879 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:13,448 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:13,878 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:14,882 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:14,928 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:15,885 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:16,886 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:17,882 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:18,882 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:19,887 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:20,803 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:20,886 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:21,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:22,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:23,888 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:24,451 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:11:24,892 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:25,115 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:11:25,115 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:11:25,116 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:11:25,452 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:25,452 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:11:25,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:26,281 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:26,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:27,453 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:27,794 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:11:27,794 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:11:27,894 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:28,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:29,453 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:29,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:30,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:31,454 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:31,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:32,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:33,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:34,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:35,889 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:36,802 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:36,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:37,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:38,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:39,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:40,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:41,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:42,468 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:11:42,468 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:42,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:11:42,793 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:11:42,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:43,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:44,890 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:45,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:46,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:47,802 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:47,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:48,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:49,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:50,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:51,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:52,802 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:52,891 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:53,462 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:53,892 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:54,892 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:55,900 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:56,402 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:11:56,402 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 with policy now 2025-01-06 17:11:56,404 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:11:56,405 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:11:56,405 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:11:56,405 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:11:56,463 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:11:56,463 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 2025-01-06 17:11:56,463 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval 2025-01-06 17:11:56,669 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/3a2q13s4-media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 2025-01-06 17:11:56,896 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:56,931 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/y3uhn7yv-media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 2025-01-06 17:11:57,463 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:11:57,793 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:11:57,793 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:11:57,884 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:11:57,897 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:58,900 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:11:59,909 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:00,914 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:01,465 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:01,911 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:02,917 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:03,465 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:03,812 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:03,920 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:04,919 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:05,466 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:05,917 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:06,917 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:07,925 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:08,818 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:08,924 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:09,923 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:10,935 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:11,933 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:12,469 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:12:12,802 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 17:12:12,802 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: stop_status 2025-01-06 17:12:12,927 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:13,170 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:12:13,170 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:13,172 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:13,470 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:12:13,470 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:12:13,560 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 17:12:13,560 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 17:12:13,724 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/t0smfsyv-events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:12:13,761 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:12:13,921 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:14,115 INFO wandb-upload_0:2085739 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpu2qocgvx 2025-01-06 17:12:14,150 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 17:12:14,159 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 17:12:14,470 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:12:14,470 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json 2025-01-06 17:12:14,470 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json 2025-01-06 17:12:14,471 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media 2025-01-06 17:12:14,471 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table 2025-01-06 17:12:14,471 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table/eval 2025-01-06 17:12:14,528 INFO SenderThread:2085739 [sender.py:send_request_log_artifact():1525] logged artifact run-8ycpgp3n-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjI0MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUwODU0NA==', 'latestArtifact': None}} 2025-01-06 17:12:14,528 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:12:14,528 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json with policy now 2025-01-06 17:12:14,528 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:14,529 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:12:14,529 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,529 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,530 DEBUG SenderThread:2085739 [sender.py:send():379] send: telemetry 2025-01-06 17:12:14,530 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,533 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,534 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,534 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,535 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,535 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,536 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,537 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,537 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,538 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,538 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,539 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,539 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,540 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,541 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,541 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,542 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,542 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,543 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,543 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,544 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:14,544 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:14,545 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 17:12:14,724 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/fscaesmc-media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json 2025-01-06 17:12:14,743 INFO wandb-upload_2:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/en3kdbz6-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json 2025-01-06 17:12:14,917 INFO wandb-upload_0:2085739 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpw5ug3ik4 2025-01-06 17:12:14,921 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:14,981 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/w01m86g1-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json 2025-01-06 17:12:15,282 INFO SenderThread:2085739 [sender.py:send_request_log_artifact():1525] logged artifact run-8ycpgp3n-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjI0Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUwODU0Nw==', 'latestArtifact': None}} 2025-01-06 17:12:15,282 DEBUG SenderThread:2085739 [sender.py:send():379] send: files 2025-01-06 17:12:15,282 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json with policy now 2025-01-06 17:12:15,282 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:12:15,282 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:15,283 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:15,471 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:15,471 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:12:15,513 INFO wandb-upload_2:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/7kw87lln-media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json 2025-01-06 17:12:15,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:16,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:17,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:18,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:19,812 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:19,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:20,922 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:21,706 DEBUG SenderThread:2085739 [sender.py:send():379] send: exit 2025-01-06 17:12:21,706 INFO SenderThread:2085739 [sender.py:send_exit():586] handling exit code: 0 2025-01-06 17:12:21,706 INFO SenderThread:2085739 [sender.py:send_exit():588] handling runtime: 279 2025-01-06 17:12:21,706 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:21,706 INFO SenderThread:2085739 [sender.py:send_exit():594] send defer 2025-01-06 17:12:21,706 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:21,706 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-06 17:12:21,707 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:21,707 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-06 17:12:21,707 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 1 2025-01-06 17:12:21,707 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:21,707 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-06 17:12:21,707 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:21,707 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-06 17:12:21,707 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 2 2025-01-06 17:12:21,707 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:21,707 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-06 17:12:21,707 INFO HandlerThread:2085739 [system_monitor.py:finish():203] Stopping system monitor 2025-01-06 17:12:21,707 DEBUG SystemMonitor:2085739 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-06 17:12:21,707 DEBUG SystemMonitor:2085739 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-06 17:12:21,708 INFO HandlerThread:2085739 [interfaces.py:finish():200] Joined cpu monitor 2025-01-06 17:12:21,709 INFO HandlerThread:2085739 [interfaces.py:finish():200] Joined disk monitor 2025-01-06 17:12:21,760 INFO HandlerThread:2085739 [interfaces.py:finish():200] Joined gpu monitor 2025-01-06 17:12:21,760 INFO HandlerThread:2085739 [interfaces.py:finish():200] Joined memory monitor 2025-01-06 17:12:21,761 INFO HandlerThread:2085739 [interfaces.py:finish():200] Joined network monitor 2025-01-06 17:12:21,761 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:21,761 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-06 17:12:21,761 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 3 2025-01-06 17:12:21,761 DEBUG SenderThread:2085739 [sender.py:send():379] send: stats 2025-01-06 17:12:21,761 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:21,762 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-06 17:12:21,762 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:21,762 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-06 17:12:21,762 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 4 2025-01-06 17:12:21,762 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:21,762 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-06 17:12:22,473 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:12:23,474 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:28,178 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:28,178 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:28,179 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: keepalive 2025-01-06 17:12:28,180 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:28,180 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-06 17:12:28,180 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 5 2025-01-06 17:12:28,180 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:28,180 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:28,180 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-06 17:12:28,180 DEBUG SenderThread:2085739 [sender.py:send():379] send: history 2025-01-06 17:12:28,181 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: summary_record 2025-01-06 17:12:28,181 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:28,181 DEBUG SenderThread:2085739 [sender.py:send():379] send: summary 2025-01-06 17:12:28,185 INFO SenderThread:2085739 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 17:12:28,185 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:28,185 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-06 17:12:28,185 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 6 2025-01-06 17:12:28,185 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:28,185 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-06 17:12:28,186 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:28,186 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-06 17:12:28,190 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 17:12:28,299 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 7 2025-01-06 17:12:28,299 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:28,299 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-06 17:12:28,299 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:28,299 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-06 17:12:28,476 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/config.yaml 2025-01-06 17:12:28,476 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:12:28,707 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:30,823 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 8 2025-01-06 17:12:30,823 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:30,823 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:30,824 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-06 17:12:30,824 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:30,824 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-06 17:12:30,824 INFO SenderThread:2085739 [job_builder.py:build():440] Attempting to build job artifact 2025-01-06 17:12:30,826 INFO SenderThread:2085739 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-06 17:12:30,828 INFO SenderThread:2085739 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-06 17:12:30,838 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 9 2025-01-06 17:12:30,839 DEBUG SenderThread:2085739 [sender.py:send():379] send: artifact 2025-01-06 17:12:30,839 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:30,842 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-06 17:12:31,327 INFO wandb-upload_0:2085739 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpn6ah3rw0 2025-01-06 17:12:31,327 INFO wandb-upload_1:2085739 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpros8k_3v 2025-01-06 17:12:31,477 INFO Thread-12 :2085739 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:31,491 INFO wandb-upload_2:2085739 [upload_job.py:push():88] Uploaded file /tmp/tmp3o0ybe06/wandb-job.json 2025-01-06 17:12:31,708 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:32,044 INFO SenderThread:2085739 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjU3MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTg5Nzc3Mg==', 'versionIndex': 421}}} 2025-01-06 17:12:32,044 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:32,044 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-06 17:12:32,044 INFO SenderThread:2085739 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-06 17:12:32,478 INFO SenderThread:2085739 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files 2025-01-06 17:12:32,478 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/config.yaml config.yaml 2025-01-06 17:12:32,478 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json wandb-summary.json 2025-01-06 17:12:32,478 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/requirements.txt requirements.txt 2025-01-06 17:12:32,479 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-metadata.json wandb-metadata.json 2025-01-06 17:12:32,479 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:12:32,481 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log output.log 2025-01-06 17:12:32,483 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/diff.patch diff.patch 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 media/videos/eval/1_eval_1_ca2cbfc725296a7e5993.mp4 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 media/videos/eval/0_eval_0_6b356ff1064e631440b3.mp4 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 media/videos/eval/4_eval_4_195f521b8e7cf2a9579a.mp4 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 media/videos/eval/2_eval_2_ff803fd511eb33dc17ee.mp4 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 media/videos/eval/3_eval_3_509210f2f4eb425aa951.mp4 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json media/table/eval/total_success_rate_ci_6_f0bc1a89c9e598fde9e4.table.json 2025-01-06 17:12:32,484 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 10 2025-01-06 17:12:32,485 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:32,485 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:32,485 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-06 17:12:32,488 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:32,488 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-06 17:12:32,488 INFO SenderThread:2085739 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 17:12:32,657 INFO wandb-upload_0:2085739 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/config.yaml 2025-01-06 17:12:32,675 INFO wandb-upload_2:2085739 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/requirements.txt 2025-01-06 17:12:32,679 INFO wandb-upload_1:2085739 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/wandb-summary.json 2025-01-06 17:12:32,708 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:32,709 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:32,710 INFO wandb-upload_4:2085739 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/files/output.log 2025-01-06 17:12:32,787 INFO wandb-upload_3:2085739 [upload_job.py:push():130] Uploaded file /tmp/tmpae5la3hbwandb/yx6fcvjx-events.out.tfevents.1736201262.leonmkim-ROG-Strix-G15CS-G15CS.2085574.0 2025-01-06 17:12:32,988 INFO Thread-11 (_thread_body):2085739 [sender.py:transition_state():614] send defer: 11 2025-01-06 17:12:32,988 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:32,988 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-06 17:12:32,988 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:32,988 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-06 17:12:32,988 INFO SenderThread:2085739 [file_pusher.py:join():175] waiting for file pusher 2025-01-06 17:12:32,990 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 12 2025-01-06 17:12:32,990 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:32,990 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-06 17:12:32,990 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:32,990 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-06 17:12:32,990 INFO SenderThread:2085739 [file_stream.py:finish():601] file stream finish called 2025-01-06 17:12:33,184 INFO SenderThread:2085739 [file_stream.py:finish():605] file stream finish is done 2025-01-06 17:12:33,184 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 13 2025-01-06 17:12:33,184 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:33,184 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-06 17:12:33,184 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:33,184 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-06 17:12:33,184 INFO SenderThread:2085739 [sender.py:transition_state():614] send defer: 14 2025-01-06 17:12:33,185 DEBUG SenderThread:2085739 [sender.py:send():379] send: final 2025-01-06 17:12:33,185 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: defer 2025-01-06 17:12:33,185 INFO HandlerThread:2085739 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-06 17:12:33,185 DEBUG SenderThread:2085739 [sender.py:send():379] send: footer 2025-01-06 17:12:33,185 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: defer 2025-01-06 17:12:33,185 INFO SenderThread:2085739 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-06 17:12:33,185 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:33,186 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:33,186 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: server_info 2025-01-06 17:12:33,186 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 17:12:33,187 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: get_summary 2025-01-06 17:12:33,187 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: server_info 2025-01-06 17:12:33,187 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-06 17:12:33,189 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 17:12:33,257 DEBUG SenderThread:2085739 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 17:12:33,258 INFO MainThread:2085739 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-06 17:12:33,258 INFO MainThread:2085739 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-06 17:12:33,258 INFO MainThread:2085739 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-06 17:12:33,259 DEBUG HandlerThread:2085739 [handler.py:handle_request():158] handle_request: shutdown 2025-01-06 17:12:33,259 INFO HandlerThread:2085739 [handler.py:finish():882] shutting down handler 2025-01-06 17:12:34,186 INFO WriterThread:2085739 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/8ycpgp3n/wandb/run-20250106_170742-8ycpgp3n/run-8ycpgp3n.wandb 2025-01-06 17:12:34,258 INFO SenderThread:2085739 [sender.py:finish():1615] shutting down sender 2025-01-06 17:12:34,258 INFO SenderThread:2085739 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 17:12:34,258 INFO SenderThread:2085739 [file_pusher.py:join():175] waiting for file pusher