2025-01-06 20:45:11,469 INFO StreamThr :2114455 [internal.py:wandb_internal():85] W&B internal server running at pid: 2114455, started at: 2025-01-06 20:45:11.469236 2025-01-06 20:45:11,470 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status 2025-01-06 20:45:11,483 INFO WriterThread:2114455 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/run-58wmqlmd.wandb 2025-01-06 20:45:11,483 DEBUG SenderThread:2114455 [sender.py:send():379] send: header 2025-01-06 20:45:11,491 DEBUG SenderThread:2114455 [sender.py:send():379] send: run 2025-01-06 20:45:11,789 INFO SenderThread:2114455 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files 2025-01-06 20:45:11,789 INFO SenderThread:2114455 [sender.py:_start_run_threads():1188] run started: 58wmqlmd with start time 1736214311.470127 2025-01-06 20:45:11,805 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: check_version 2025-01-06 20:45:11,806 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: check_version 2025-01-06 20:45:11,862 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: run_start 2025-01-06 20:45:11,866 DEBUG HandlerThread:2114455 [system_info.py:__init__():26] System info init 2025-01-06 20:45:11,866 DEBUG HandlerThread:2114455 [system_info.py:__init__():41] System info init done 2025-01-06 20:45:11,866 INFO HandlerThread:2114455 [system_monitor.py:start():194] Starting system monitor 2025-01-06 20:45:11,866 INFO SystemMonitor:2114455 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-06 20:45:11,867 INFO HandlerThread:2114455 [system_monitor.py:probe():214] Collecting system info 2025-01-06 20:45:11,868 INFO SystemMonitor:2114455 [interfaces.py:start():188] Started cpu monitoring 2025-01-06 20:45:11,868 INFO SystemMonitor:2114455 [interfaces.py:start():188] Started disk monitoring 2025-01-06 20:45:11,869 INFO SystemMonitor:2114455 [interfaces.py:start():188] Started gpu monitoring 2025-01-06 20:45:11,869 INFO SystemMonitor:2114455 [interfaces.py:start():188] Started memory monitoring 2025-01-06 20:45:11,870 INFO SystemMonitor:2114455 [interfaces.py:start():188] Started network monitoring 2025-01-06 20:45:11,879 DEBUG HandlerThread:2114455 [system_info.py:probe():152] Probing system 2025-01-06 20:45:11,880 DEBUG HandlerThread:2114455 [system_info.py:_probe_git():137] Probing git 2025-01-06 20:45:11,885 DEBUG HandlerThread:2114455 [system_info.py:_probe_git():145] Probing git done 2025-01-06 20:45:11,885 DEBUG HandlerThread:2114455 [system_info.py:probe():200] Probing system done 2025-01-06 20:45:11,885 DEBUG HandlerThread:2114455 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T01:45:11.879429', 'startedAt': '2025-01-07T01:45:11.461617', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.469375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.904, 'min': 800.0, 'max': 4700.0}, {'current': 4499.924, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.914, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 518.7194671630859}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2025-01-06 20:45:11,885 INFO HandlerThread:2114455 [system_monitor.py:probe():224] Finished collecting system info 2025-01-06 20:45:11,885 INFO HandlerThread:2114455 [system_monitor.py:probe():227] Publishing system info 2025-01-06 20:45:11,889 DEBUG HandlerThread:2114455 [system_info.py:_save_code():44] Saving code 2025-01-06 20:45:11,891 DEBUG HandlerThread:2114455 [system_info.py:_save_code():67] Saving code done 2025-01-06 20:45:11,891 DEBUG HandlerThread:2114455 [system_info.py:_save_patches():84] Saving git patches 2025-01-06 20:45:11,952 DEBUG HandlerThread:2114455 [system_info.py:_save_patches():126] Saving git patches done 2025-01-06 20:45:11,952 INFO HandlerThread:2114455 [system_monitor.py:probe():229] Finished publishing system info 2025-01-06 20:45:11,956 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:45:11,956 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-06 20:45:11,956 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2025-01-06 20:45:11,956 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file diff.patch with policy now 2025-01-06 20:45:12,182 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: python_packages 2025-01-06 20:45:12,182 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: python_packages 2025-01-06 20:45:12,182 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:45:12,183 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:45:12,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:12,185 INFO HandlerThread:2114455 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/tb" save: true } control { relay_id: "e0935560bead4ef6a2b545171ed29f67" } _info { stream_id: "58wmqlmd" } 2025-01-06 20:45:12,185 DEBUG HandlerThread:2114455 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2025-01-06 20:45:12,219 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/k35u63es-code/FISH/eval_robot.py 2025-01-06 20:45:12,241 INFO wandb-upload_2:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/luw4jhex-diff.patch 2025-01-06 20:45:12,241 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/9tpw7i1e-wandb-metadata.json 2025-01-06 20:45:12,286 DEBUG SenderThread:2114455 [sender.py:send():379] send: telemetry 2025-01-06 20:45:12,286 DEBUG SenderThread:2114455 [sender.py:send():379] send: tbrecord 2025-01-06 20:45:12,287 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:45:12,287 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 with policy live 2025-01-06 20:45:12,507 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/gnzzwpnd-events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:45:12,790 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/code/FISH/eval_robot.py 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-metadata.json 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/requirements.txt 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/diff.patch 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/code/FISH 2025-01-06 20:45:12,791 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/code 2025-01-06 20:45:13,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:14,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:14,790 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:15,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:16,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:17,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:17,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:18,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:19,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:20,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:21,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:22,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:22,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:23,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:24,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:25,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:26,185 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:27,182 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:45:27,183 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:45:27,225 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:27,305 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:28,154 DEBUG SenderThread:2114455 [sender.py:send():379] send: config 2025-01-06 20:45:28,197 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:29,201 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:30,185 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:30,794 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:31,185 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:32,185 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:32,794 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:32,881 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:33,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:34,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:34,795 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:35,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:36,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:36,796 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:37,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:38,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:38,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:39,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:40,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:41,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:42,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:45:42,183 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:45:42,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:42,797 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:43,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:44,156 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:44,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:44,798 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:44,798 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/config.yaml 2025-01-06 20:45:45,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:46,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:46,799 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:45:47,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:48,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:49,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:50,184 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:50,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:51,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:52,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:53,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:54,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:55,185 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:45:55,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:56,190 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:57,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:45:57,183 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:45:57,225 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:58,190 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:45:59,190 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:00,197 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:00,275 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:01,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:02,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:03,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:04,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:05,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:06,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:06,192 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:07,192 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:08,193 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:09,194 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:10,202 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:11,202 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:11,870 DEBUG SystemMonitor:2114455 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-06 20:46:11,871 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:46:11,872 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:12,183 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:46:12,183 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:46:12,225 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:13,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:13,773 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:46:13,773 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 with policy now 2025-01-06 20:46:13,774 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:46:13,774 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:46:13,774 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:46:13,775 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:46:13,805 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:46:13,806 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 2025-01-06 20:46:13,806 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos 2025-01-06 20:46:13,806 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media 2025-01-06 20:46:13,806 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval 2025-01-06 20:46:14,028 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/y1o2ohig-media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 2025-01-06 20:46:14,204 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:14,303 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/7nsgj814-media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 2025-01-06 20:46:14,806 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:15,204 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:16,205 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:17,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:17,205 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:18,205 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:18,807 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:19,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:20,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:20,808 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:21,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:22,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:22,730 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:22,809 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:46:23,049 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/horu78vb-events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:46:23,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:24,206 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:24,809 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:25,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:26,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:27,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:46:27,186 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:46:27,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:28,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:28,345 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:28,810 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:29,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:30,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:31,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:32,207 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:33,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:34,188 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:34,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:35,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:36,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:37,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:38,210 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:39,188 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:39,211 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:40,211 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:41,211 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:41,871 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:46:42,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:46:42,186 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:46:42,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:43,211 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:44,212 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:44,320 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:45,212 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:46,212 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:47,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:48,214 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:49,214 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:50,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:50,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:51,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:52,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:53,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:54,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:54,819 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:54,941 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:46:54,941 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 with policy now 2025-01-06 20:46:54,942 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:46:54,942 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:46:54,942 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:46:54,943 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:46:55,173 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/8e360ud1-media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 2025-01-06 20:46:55,215 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:55,819 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:46:55,820 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 2025-01-06 20:46:55,820 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval 2025-01-06 20:46:55,959 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:46:56,098 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/m1d3ahwf-media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 2025-01-06 20:46:56,218 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:56,820 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:46:57,195 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:46:57,196 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:46:57,237 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:58,222 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:46:59,222 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:00,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:00,821 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:01,233 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:01,396 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:02,225 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:02,822 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:03,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:04,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:04,823 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:47:05,225 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:05,687 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:47:05,687 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:47:05,687 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:47:05,823 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:47:06,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:06,729 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:06,823 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:07,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:08,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:09,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:10,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:10,825 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:11,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:11,872 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:47:11,872 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:12,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:47:12,186 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:47:12,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:13,226 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:14,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:15,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:16,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:17,189 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:17,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:18,227 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:19,228 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:20,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:21,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:22,190 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:22,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:23,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:24,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:25,229 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:26,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:27,186 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:47:27,187 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:47:27,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:27,254 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:28,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:29,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:30,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:31,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:32,230 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:33,191 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:33,231 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:34,231 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:35,231 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:36,232 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:36,689 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:47:36,689 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:47:36,690 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 with policy now 2025-01-06 20:47:36,690 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:47:36,690 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:47:36,690 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:47:36,833 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:36,833 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:47:36,834 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 2025-01-06 20:47:36,834 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval 2025-01-06 20:47:36,926 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/0hvd5p73-media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 2025-01-06 20:47:37,198 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/pcqbtz2z-media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 2025-01-06 20:47:37,241 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:38,194 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:38,237 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:38,834 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:39,234 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:40,236 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:41,237 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:41,872 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:47:42,200 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:47:42,200 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:47:42,241 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:42,835 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:43,235 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:44,145 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:44,240 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:44,836 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:45,239 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:46,238 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:47,242 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:48,242 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:48,837 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:47:49,083 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/kwsd77pi-events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:47:49,238 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:49,443 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:47:49,443 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:47:49,443 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:49,443 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:47:49,838 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:47:50,241 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:50,838 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:51,245 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:52,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:53,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:54,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:54,534 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:47:54,839 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:55,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:56,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:56,840 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:47:57,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:47:57,187 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:47:57,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:58,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:47:59,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:00,250 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:00,386 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:01,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:02,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:03,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:04,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:05,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:05,534 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:06,387 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:06,843 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:07,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:08,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:09,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:10,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:10,581 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:10,845 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:11,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:11,873 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:48:12,187 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:48:12,187 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:48:12,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:13,388 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:14,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:15,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:16,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:16,581 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:16,847 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:17,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:18,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:19,389 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:20,392 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:20,848 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:20,909 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:48:20,909 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 with policy now 2025-01-06 20:48:20,913 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:48:20,913 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:48:20,914 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:48:20,914 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:48:21,134 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/f12j9vht-media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 2025-01-06 20:48:21,399 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:21,848 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:48:21,849 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 2025-01-06 20:48:21,849 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval 2025-01-06 20:48:21,936 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:22,137 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/65uaq9mh-media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 2025-01-06 20:48:22,400 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:22,849 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:23,400 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:23,849 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:24,398 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:25,400 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:26,398 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:26,850 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:27,204 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:27,205 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:48:27,205 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:48:27,403 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:28,402 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:28,851 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:29,401 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:30,401 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:31,401 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:32,409 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:33,203 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:33,406 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:34,407 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:35,404 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:35,854 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:48:36,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:36,499 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:48:36,499 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:48:36,500 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:48:36,854 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:36,854 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:48:37,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:38,203 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:38,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:38,855 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:39,407 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:40,407 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:40,856 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:48:41,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:41,873 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:48:42,200 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:48:42,200 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:48:42,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:43,330 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:43,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:44,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:45,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:46,408 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:47,411 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:48,411 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:49,202 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:49,411 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:50,411 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:51,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:52,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:53,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:54,203 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:54,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:55,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:56,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:57,202 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:48:57,202 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:48:57,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:58,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:48:59,344 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:48:59,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:00,412 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:01,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:02,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:03,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:04,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:04,821 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:04,865 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:05,413 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:06,418 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:07,425 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:07,768 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:49:07,768 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 with policy now 2025-01-06 20:49:07,768 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:49:07,769 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:49:07,769 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:07,769 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:07,866 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:07,866 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 2025-01-06 20:49:07,866 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval 2025-01-06 20:49:07,996 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/y3bsjbqj-media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 2025-01-06 20:49:08,268 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/8hu5vf2q-media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 2025-01-06 20:49:08,421 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:08,866 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:09,423 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:09,867 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:10,424 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:10,797 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:11,427 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:11,874 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:49:12,213 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: stop_status 2025-01-06 20:49:12,213 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: stop_status 2025-01-06 20:49:12,429 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:12,868 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:13,437 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:14,429 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:15,429 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:16,357 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:16,437 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:16,869 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:17,431 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:18,435 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:19,432 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:20,436 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:21,445 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:22,214 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:22,441 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:22,555 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:49:22,555 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:22,555 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:22,783 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 20:49:22,783 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 20:49:22,871 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:22,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:49:22,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json 2025-01-06 20:49:22,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media 2025-01-06 20:49:22,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval 2025-01-06 20:49:22,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table 2025-01-06 20:49:22,966 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:49:23,053 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/wljwusm6-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json 2025-01-06 20:49:23,121 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/g0nep7sp-events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:49:23,297 INFO wandb-upload_2:2114455 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpox_lvbf5 2025-01-06 20:49:23,330 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: log_artifact 2025-01-06 20:49:23,339 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: partial_history 2025-01-06 20:49:23,435 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:23,668 INFO SenderThread:2114455 [sender.py:send_request_log_artifact():1525] logged artifact run-58wmqlmd-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjMyOTE3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY0ODQwMw==', 'latestArtifact': None}} 2025-01-06 20:49:23,669 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:49:23,669 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now 2025-01-06 20:49:23,669 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:49:23,669 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,670 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,670 DEBUG SenderThread:2114455 [sender.py:send():379] send: telemetry 2025-01-06 20:49:23,670 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,670 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,671 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,671 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,671 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,673 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,673 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,673 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,674 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,674 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,674 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,674 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,674 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,675 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,675 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,675 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,675 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,676 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,676 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,676 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,676 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:23,677 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:23,677 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: log_artifact 2025-01-06 20:49:23,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:23,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:49:23,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json 2025-01-06 20:49:23,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval 2025-01-06 20:49:23,907 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/i0u49oyc-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json 2025-01-06 20:49:24,062 INFO wandb-upload_2:2114455 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpm48d6fxb 2025-01-06 20:49:24,064 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/nradoyfc-media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json 2025-01-06 20:49:24,435 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:24,504 INFO SenderThread:2114455 [sender.py:send_request_log_artifact():1525] logged artifact run-58wmqlmd-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjMyOTIwMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY0ODQwNA==', 'latestArtifact': None}} 2025-01-06 20:49:24,504 DEBUG SenderThread:2114455 [sender.py:send():379] send: files 2025-01-06 20:49:24,504 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json with policy now 2025-01-06 20:49:24,505 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:49:24,505 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:24,505 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:24,626 DEBUG SenderThread:2114455 [sender.py:send():379] send: exit 2025-01-06 20:49:24,626 INFO SenderThread:2114455 [sender.py:send_exit():586] handling exit code: 0 2025-01-06 20:49:24,626 INFO SenderThread:2114455 [sender.py:send_exit():588] handling runtime: 252 2025-01-06 20:49:24,626 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:24,626 INFO SenderThread:2114455 [sender.py:send_exit():594] send defer 2025-01-06 20:49:24,626 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:24,626 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-06 20:49:24,627 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:24,627 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-06 20:49:24,627 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 1 2025-01-06 20:49:24,627 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:24,627 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-06 20:49:24,627 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:24,627 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-06 20:49:24,627 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 2 2025-01-06 20:49:24,627 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:24,627 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-06 20:49:24,627 INFO HandlerThread:2114455 [system_monitor.py:finish():203] Stopping system monitor 2025-01-06 20:49:24,628 DEBUG SystemMonitor:2114455 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-06 20:49:24,628 DEBUG SystemMonitor:2114455 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-06 20:49:24,629 INFO HandlerThread:2114455 [interfaces.py:finish():200] Joined cpu monitor 2025-01-06 20:49:24,629 INFO HandlerThread:2114455 [interfaces.py:finish():200] Joined disk monitor 2025-01-06 20:49:24,680 INFO HandlerThread:2114455 [interfaces.py:finish():200] Joined gpu monitor 2025-01-06 20:49:24,680 INFO HandlerThread:2114455 [interfaces.py:finish():200] Joined memory monitor 2025-01-06 20:49:24,681 INFO HandlerThread:2114455 [interfaces.py:finish():200] Joined network monitor 2025-01-06 20:49:24,681 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:24,681 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-06 20:49:24,681 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 3 2025-01-06 20:49:24,681 DEBUG SenderThread:2114455 [sender.py:send():379] send: stats 2025-01-06 20:49:24,681 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:24,681 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-06 20:49:24,682 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:24,682 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-06 20:49:24,682 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 4 2025-01-06 20:49:24,682 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:24,682 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-06 20:49:24,761 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/54bw3s1j-media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json 2025-01-06 20:49:24,872 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:24,873 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:26,873 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:31,561 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:31,561 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:31,562 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: keepalive 2025-01-06 20:49:31,563 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:31,563 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-06 20:49:31,563 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 5 2025-01-06 20:49:31,563 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:31,563 DEBUG SenderThread:2114455 [sender.py:send():379] send: history 2025-01-06 20:49:31,563 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: summary_record 2025-01-06 20:49:31,564 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:31,564 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-06 20:49:31,564 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:31,564 DEBUG SenderThread:2114455 [sender.py:send():379] send: summary 2025-01-06 20:49:31,568 INFO SenderThread:2114455 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-06 20:49:31,568 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:31,568 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-06 20:49:31,568 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 6 2025-01-06 20:49:31,568 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:31,568 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-06 20:49:31,568 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:31,568 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-06 20:49:31,572 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: status_report 2025-01-06 20:49:31,627 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:31,695 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 7 2025-01-06 20:49:31,695 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:31,695 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:31,695 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-06 20:49:31,695 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:31,695 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-06 20:49:31,875 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/config.yaml 2025-01-06 20:49:31,875 INFO Thread-12 :2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:32,627 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:32,961 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 8 2025-01-06 20:49:32,961 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:32,962 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:32,962 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-06 20:49:32,962 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:32,962 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-06 20:49:32,962 INFO SenderThread:2114455 [job_builder.py:build():440] Attempting to build job artifact 2025-01-06 20:49:32,964 INFO SenderThread:2114455 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-06 20:49:32,966 INFO SenderThread:2114455 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-06 20:49:32,975 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 9 2025-01-06 20:49:32,975 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:32,976 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-06 20:49:32,976 DEBUG SenderThread:2114455 [sender.py:send():379] send: artifact 2025-01-06 20:49:33,181 INFO SenderThread:2114455 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTY1Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} 2025-01-06 20:49:33,181 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:33,181 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-06 20:49:33,181 INFO SenderThread:2114455 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-06 20:49:33,628 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:33,876 INFO SenderThread:2114455 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:33,876 INFO SenderThread:2114455 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/config.yaml config.yaml 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json wandb-summary.json 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/requirements.txt requirements.txt 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-metadata.json wandb-metadata.json 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log output.log 2025-01-06 20:49:33,877 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/diff.patch diff.patch 2025-01-06 20:49:33,879 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2025-01-06 20:49:33,885 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 media/videos/eval/4_eval_4_760e7eee3264b80c008d.mp4 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 media/videos/eval/0_eval_0_d0d9ac7b828987714ef0.mp4 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 media/videos/eval/2_eval_2_25649ac765ce7bb6dc60.mp4 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 media/videos/eval/3_eval_3_f942d3972a955727292e.mp4 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 media/videos/eval/1_eval_1_a3c1331fe3dbbcb17ff3.mp4 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json media/table/eval/total_success_rate_ci_6_c68d11fd8ad6b790b7a9.table.json 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json 2025-01-06 20:49:33,886 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 10 2025-01-06 20:49:33,886 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:33,886 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:33,886 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-06 20:49:33,887 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:33,887 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-06 20:49:33,887 INFO SenderThread:2114455 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 20:49:34,071 INFO wandb-upload_0:2114455 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/config.yaml 2025-01-06 20:49:34,076 INFO wandb-upload_1:2114455 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/requirements.txt 2025-01-06 20:49:34,097 INFO wandb-upload_2:2114455 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/wandb-summary.json 2025-01-06 20:49:34,139 INFO wandb-upload_4:2114455 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/files/output.log 2025-01-06 20:49:34,147 INFO wandb-upload_3:2114455 [upload_job.py:push():130] Uploaded file /tmp/tmpzv1tdaddwandb/mkbs4unt-events.out.tfevents.1736214312.leonmkim-ROG-Strix-G15CS-G15CS.2114277.0 2025-01-06 20:49:34,348 INFO Thread-11 (_thread_body):2114455 [sender.py:transition_state():614] send defer: 11 2025-01-06 20:49:34,348 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:34,348 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-06 20:49:34,348 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:34,348 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-06 20:49:34,349 INFO SenderThread:2114455 [file_pusher.py:join():175] waiting for file pusher 2025-01-06 20:49:34,350 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 12 2025-01-06 20:49:34,350 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:34,350 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-06 20:49:34,350 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:34,350 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-06 20:49:34,350 INFO SenderThread:2114455 [file_stream.py:finish():601] file stream finish called 2025-01-06 20:49:34,549 INFO SenderThread:2114455 [file_stream.py:finish():605] file stream finish is done 2025-01-06 20:49:34,549 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 13 2025-01-06 20:49:34,549 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:34,549 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-06 20:49:34,550 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:34,550 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-06 20:49:34,550 INFO SenderThread:2114455 [sender.py:transition_state():614] send defer: 14 2025-01-06 20:49:34,550 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: defer 2025-01-06 20:49:34,550 INFO HandlerThread:2114455 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-06 20:49:34,550 DEBUG SenderThread:2114455 [sender.py:send():379] send: final 2025-01-06 20:49:34,550 DEBUG SenderThread:2114455 [sender.py:send():379] send: footer 2025-01-06 20:49:34,550 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: defer 2025-01-06 20:49:34,550 INFO SenderThread:2114455 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-06 20:49:34,551 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:34,551 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: server_info 2025-01-06 20:49:34,551 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:34,551 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: server_info 2025-01-06 20:49:34,552 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-06 20:49:34,553 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: get_summary 2025-01-06 20:49:34,554 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-06 20:49:34,555 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-06 20:49:34,630 DEBUG SenderThread:2114455 [sender.py:send_request():406] send_request: poll_exit 2025-01-06 20:49:34,631 INFO MainThread:2114455 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2025-01-06 20:49:34,631 INFO MainThread:2114455 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2025-01-06 20:49:34,631 INFO MainThread:2114455 [wandb_run.py:_footer_sync_info():3975] logging synced files 2025-01-06 20:49:34,631 DEBUG HandlerThread:2114455 [handler.py:handle_request():158] handle_request: shutdown 2025-01-06 20:49:34,632 INFO HandlerThread:2114455 [handler.py:finish():882] shutting down handler 2025-01-06 20:49:35,555 INFO WriterThread:2114455 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/58wmqlmd/wandb/run-20250106_204511-58wmqlmd/run-58wmqlmd.wandb 2025-01-06 20:49:35,631 INFO SenderThread:2114455 [sender.py:finish():1615] shutting down sender 2025-01-06 20:49:35,631 INFO SenderThread:2114455 [file_pusher.py:finish():169] shutting down file pusher 2025-01-06 20:49:35,631 INFO SenderThread:2114455 [file_pusher.py:join():175] waiting for file pusher