2024-12-17 22:03:30,880 INFO StreamThr :81075 [internal.py:wandb_internal():85] W&B internal server running at pid: 81075, started at: 2024-12-17 22:03:30.880596 2024-12-17 22:03:30,881 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status 2024-12-17 22:03:30,892 INFO WriterThread:81075 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/run-o350ecc4.wandb 2024-12-17 22:03:30,893 DEBUG SenderThread:81075 [sender.py:send():379] send: header 2024-12-17 22:03:30,900 DEBUG SenderThread:81075 [sender.py:send():379] send: run 2024-12-17 22:03:31,120 INFO SenderThread:81075 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files 2024-12-17 22:03:31,120 INFO SenderThread:81075 [sender.py:_start_run_threads():1188] run started: o350ecc4 with start time 1734491010.881591 2024-12-17 22:03:31,137 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: check_version 2024-12-17 22:03:31,137 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: check_version 2024-12-17 22:03:31,183 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: run_start 2024-12-17 22:03:31,187 DEBUG HandlerThread:81075 [system_info.py:__init__():26] System info init 2024-12-17 22:03:31,187 DEBUG HandlerThread:81075 [system_info.py:__init__():41] System info init done 2024-12-17 22:03:31,187 INFO HandlerThread:81075 [system_monitor.py:start():194] Starting system monitor 2024-12-17 22:03:31,188 INFO SystemMonitor:81075 [system_monitor.py:_start():158] Starting system asset monitoring threads 2024-12-17 22:03:31,188 INFO HandlerThread:81075 [system_monitor.py:probe():214] Collecting system info 2024-12-17 22:03:31,189 INFO SystemMonitor:81075 [interfaces.py:start():188] Started cpu monitoring 2024-12-17 22:03:31,189 INFO SystemMonitor:81075 [interfaces.py:start():188] Started disk monitoring 2024-12-17 22:03:31,190 INFO SystemMonitor:81075 [interfaces.py:start():188] Started gpu monitoring 2024-12-17 22:03:31,190 INFO SystemMonitor:81075 [interfaces.py:start():188] Started memory monitoring 2024-12-17 22:03:31,191 INFO SystemMonitor:81075 [interfaces.py:start():188] Started network monitoring 2024-12-17 22:03:31,201 DEBUG HandlerThread:81075 [system_info.py:probe():152] Probing system 2024-12-17 22:03:31,203 DEBUG HandlerThread:81075 [system_info.py:_probe_git():137] Probing git 2024-12-17 22:03:31,208 DEBUG HandlerThread:81075 [system_info.py:_probe_git():145] Probing git done 2024-12-17 22:03:31,208 DEBUG HandlerThread:81075 [system_info.py:probe():200] Probing system done 2024-12-17 22:03:31,208 DEBUG HandlerThread:81075 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T03:03:31.201770', 'startedAt': '2024-12-18T03:03:30.869253', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4124.99825, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.986, 'min': 800.0, 'max': 4700.0}, {'current': 4499.988, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.995, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 404.623722076416}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} 2024-12-17 22:03:31,208 INFO HandlerThread:81075 [system_monitor.py:probe():224] Finished collecting system info 2024-12-17 22:03:31,208 INFO HandlerThread:81075 [system_monitor.py:probe():227] Publishing system info 2024-12-17 22:03:31,208 DEBUG HandlerThread:81075 [system_info.py:_save_code():44] Saving code 2024-12-17 22:03:31,211 DEBUG HandlerThread:81075 [system_info.py:_save_code():67] Saving code done 2024-12-17 22:03:31,211 DEBUG HandlerThread:81075 [system_info.py:_save_patches():84] Saving git patches 2024-12-17 22:03:31,278 DEBUG HandlerThread:81075 [system_info.py:_save_patches():126] Saving git patches done 2024-12-17 22:03:31,278 INFO HandlerThread:81075 [system_monitor.py:probe():229] Finished publishing system info 2024-12-17 22:03:31,282 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:03:31,282 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2024-12-17 22:03:31,282 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now 2024-12-17 22:03:31,282 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file diff.patch with policy now 2024-12-17 22:03:31,524 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: python_packages 2024-12-17 22:03:31,524 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: python_packages 2024-12-17 22:03:31,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:03:31,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:03:31,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:31,527 INFO HandlerThread:81075 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/tb" save: true } control { relay_id: "b768d79f62c44bf98e5426c1d042426b" } _info { stream_id: "o350ecc4" } 2024-12-17 22:03:31,527 DEBUG HandlerThread:81075 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml 2024-12-17 22:03:31,536 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/beyejj7z-diff.patch 2024-12-17 22:03:31,577 INFO wandb-upload_0:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/x9iz66yc-wandb-metadata.json 2024-12-17 22:03:31,587 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/lkynvg6v-code/FISH/eval_robot.py 2024-12-17 22:03:31,614 DEBUG SenderThread:81075 [sender.py:send():379] send: telemetry 2024-12-17 22:03:31,615 DEBUG SenderThread:81075 [sender.py:send():379] send: tbrecord 2024-12-17 22:03:31,615 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:03:31,615 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 with policy live 2024-12-17 22:03:31,811 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/bayjrbcz-events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:03:32,121 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:03:32,121 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/requirements.txt 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/code/FISH/eval_robot.py 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/diff.patch 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-metadata.json 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/code/FISH 2024-12-17 22:03:32,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/code 2024-12-17 22:03:32,524 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:33,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:34,120 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:34,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:35,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:36,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:03:36,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:37,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:38,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:38,527 DEBUG SenderThread:81075 [sender.py:send():379] send: config 2024-12-17 22:03:39,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:40,122 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:40,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:41,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:03:41,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:42,123 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:42,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:43,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:44,123 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:44,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:45,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:46,124 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:46,524 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:03:46,524 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:03:46,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:46,586 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:03:47,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:48,125 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:48,526 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:49,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:50,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:51,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:52,126 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:52,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:03:52,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:53,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:54,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:55,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:56,127 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:56,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:57,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:57,660 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:03:58,128 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:03:58,527 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:03:59,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:00,128 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:00,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:01,524 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:04:01,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:04:01,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:02,129 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:02,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:02,794 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:03,129 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/config.yaml 2024-12-17 22:04:03,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:04,129 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:04,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:05,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:06,130 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:06,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:07,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:07,976 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:08,130 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:08,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:09,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:10,131 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:10,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:11,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:12,132 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:12,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:13,181 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:13,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:14,132 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:14,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:15,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:16,133 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:16,524 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:04:16,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:04:16,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:17,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:18,133 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:18,368 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:18,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:19,531 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:20,134 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:20,532 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:21,120 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:04:21,120 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 with policy now 2024-12-17 22:04:21,121 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:04:21,122 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:04:21,123 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:04:21,124 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:04:21,134 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 2024-12-17 22:04:21,135 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:04:21,135 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos 2024-12-17 22:04:21,135 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval 2024-12-17 22:04:21,135 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media 2024-12-17 22:04:21,382 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/o1nho8qn-media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 2024-12-17 22:04:21,533 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:21,643 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/4osllq2q-media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 2024-12-17 22:04:22,135 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:22,532 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:23,528 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:23,533 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:24,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:25,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:26,136 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:26,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:27,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:28,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:29,214 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:29,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:30,137 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:04:30,138 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:30,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:31,191 DEBUG SystemMonitor:81075 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2024-12-17 22:04:31,191 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:04:31,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:04:31,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:04:31,565 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:32,138 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:32,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:33,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:34,139 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:34,414 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:34,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:35,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:36,140 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:36,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:37,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:38,140 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:38,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:39,431 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:39,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:40,141 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:40,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:41,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:42,142 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:42,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:43,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:44,142 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:44,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:44,627 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:45,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:46,143 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:46,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:04:46,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:04:46,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:47,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:48,144 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:48,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:49,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:49,628 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:50,144 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:50,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:51,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:52,145 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:52,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:53,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:54,146 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:54,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:54,827 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:04:55,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:56,147 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:56,537 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:57,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:58,147 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:04:58,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:59,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:04:59,839 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:00,148 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:00,539 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:01,192 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:05:01,205 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:05:01,205 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 with policy now 2024-12-17 22:05:01,206 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:05:01,209 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:05:01,209 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:05:01,209 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:05:01,492 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/igqxudol-media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 2024-12-17 22:05:01,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:05:01,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:05:01,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:02,149 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:05:02,149 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:02,149 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 2024-12-17 22:05:02,149 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval 2024-12-17 22:05:02,450 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/yn3czz1v-media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 2024-12-17 22:05:02,548 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:03,544 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:04,149 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:04,544 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:04,920 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:05,546 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:06,150 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:06,548 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:07,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:08,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:09,548 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:10,151 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:10,548 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:10,685 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:11,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:12,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:13,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:14,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:15,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:16,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:05:16,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:05:16,569 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:16,613 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:17,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:18,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:19,550 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:20,155 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:05:20,396 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/p8kp05iu-events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:05:20,550 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:20,933 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:05:20,933 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:05:20,934 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:05:21,155 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:05:21,550 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:22,156 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:22,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:22,551 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:23,551 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:24,552 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:25,552 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:26,157 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:26,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:27,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:27,624 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:28,158 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:28,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:29,554 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:30,158 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:30,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:31,192 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:05:31,525 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:05:31,525 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:05:31,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:32,159 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:32,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:32,815 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:33,553 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:34,160 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:34,554 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:35,554 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:36,161 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:36,554 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:37,556 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:37,891 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:37,919 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:05:37,919 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 with policy now 2024-12-17 22:05:37,919 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:05:37,920 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:05:37,920 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:05:37,921 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:05:38,159 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/nljzs0z6-media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 2024-12-17 22:05:38,161 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:05:38,162 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:38,162 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 2024-12-17 22:05:38,162 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval 2024-12-17 22:05:38,417 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/ddc5vchk-media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 2024-12-17 22:05:38,558 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:39,558 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:40,162 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:40,557 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:41,560 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:42,163 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:42,565 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:43,535 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:43,561 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:44,559 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:45,562 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:46,164 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:05:46,530 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:05:46,531 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:05:46,577 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:47,568 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:48,566 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:48,591 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:49,566 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:50,565 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:51,571 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:52,567 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:53,567 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:54,538 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:05:54,569 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:55,569 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:56,570 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:57,567 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:58,579 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:05:59,569 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:00,532 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:00,573 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:01,193 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:06:01,529 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:06:01,529 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:06:01,578 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:02,574 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:03,574 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:04,574 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:05,574 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:05,631 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:06,580 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:07,579 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:08,587 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:09,580 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:10,581 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:11,542 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:11,581 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:12,585 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:13,583 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:14,585 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:15,588 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:16,532 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:06:16,532 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:06:16,585 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:16,598 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:17,584 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:18,585 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:19,587 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:20,590 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:21,585 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:22,552 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:22,588 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:23,586 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:24,590 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:25,590 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:26,590 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:27,591 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:28,549 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:28,592 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:29,600 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:30,594 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:31,193 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:06:31,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:06:31,534 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:06:31,596 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:32,599 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:33,602 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:33,683 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:34,599 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:35,599 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:36,601 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:37,608 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:38,601 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:39,540 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:39,603 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:40,604 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:41,602 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:42,608 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:43,610 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:44,543 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:44,619 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:45,609 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:46,542 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:06:46,542 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:06:46,607 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:47,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:48,610 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:49,609 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:49,622 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:50,614 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:50,827 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:06:50,828 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:06:50,828 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:06:51,187 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:06:51,187 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:06:51,443 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/cme9h4m6-events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:06:51,621 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:52,187 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:06:52,614 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:53,613 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:54,612 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:54,953 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:06:55,613 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:56,188 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:06:56,612 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:57,613 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:58,189 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:06:58,614 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:59,615 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:06:59,956 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:00,190 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:00,615 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:01,194 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:07:01,531 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:07:01,532 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:07:01,614 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:02,190 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:02,615 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:03,615 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:04,191 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:04,616 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:04,973 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:05,616 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:06,192 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:06,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:07,616 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:08,193 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:08,616 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:09,616 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:09,981 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:10,193 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:10,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:11,621 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:12,194 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:12,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:13,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:14,195 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:14,618 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:15,006 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:15,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:16,196 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:16,532 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:07:16,532 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:07:16,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:17,617 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:18,197 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:18,618 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:19,618 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:20,031 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:20,198 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:20,618 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:21,620 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:22,198 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:22,621 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:22,993 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:07:22,993 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 with policy now 2024-12-17 22:07:23,010 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:07:23,010 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:07:23,010 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:07:23,011 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:07:23,199 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:07:23,199 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 2024-12-17 22:07:23,199 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval 2024-12-17 22:07:23,262 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/h6jxl0yx-media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 2024-12-17 22:07:23,530 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/245oases-media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 2024-12-17 22:07:23,623 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:24,199 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:24,626 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:25,036 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:25,631 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:26,626 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:27,629 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:28,200 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:28,633 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:29,624 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:30,201 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:30,541 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:30,642 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:31,194 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:07:31,540 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:07:31,540 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:07:31,636 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:32,645 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:33,643 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:34,644 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:35,638 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:35,696 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:36,642 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:37,640 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:38,640 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:39,645 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:40,640 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:41,548 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:41,639 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:42,642 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:43,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:44,646 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:45,645 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:46,536 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:07:46,536 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:07:46,591 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:46,651 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:47,651 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:48,207 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:07:48,646 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:48,900 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:07:48,900 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:07:48,901 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:07:49,207 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:07:49,645 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:50,208 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:50,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:51,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:51,974 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:52,646 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:53,646 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:54,209 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:54,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:55,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:56,210 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:56,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:57,050 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:07:57,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:58,210 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:07:58,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:07:59,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:00,211 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:00,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:01,195 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:08:01,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:08:01,534 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:08:01,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:02,064 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:02,212 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:02,647 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:03,648 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:04,213 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:04,648 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:05,648 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:06,213 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:06,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:07,263 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:07,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:08,214 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:08,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:09,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:10,215 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:10,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:11,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:12,216 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:12,264 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:12,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:13,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:14,216 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:14,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:15,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:16,217 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:16,534 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: stop_status 2024-12-17 22:08:16,534 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: stop_status 2024-12-17 22:08:16,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:17,450 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:17,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:18,218 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:18,650 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:19,665 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:20,109 DEBUG SenderThread:81075 [sender.py:send():379] send: files 2024-12-17 22:08:20,109 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 with policy now 2024-12-17 22:08:20,113 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: partial_history 2024-12-17 22:08:20,113 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:08:20,113 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:08:20,113 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:08:20,219 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:08:20,219 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:20,219 INFO Thread-12 :81075 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 2024-12-17 22:08:20,219 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval 2024-12-17 22:08:20,374 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/hdqfmlrp-media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 2024-12-17 22:08:20,645 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/qn7v8p40-media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 2024-12-17 22:08:20,669 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:21,665 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:22,219 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:22,561 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:22,664 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:23,666 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:24,220 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:24,671 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:25,672 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:26,676 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:27,649 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:27,675 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:28,221 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:28,672 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:29,675 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:29,781 DEBUG SenderThread:81075 [sender.py:send():379] send: exit 2024-12-17 22:08:29,781 INFO SenderThread:81075 [sender.py:send_exit():586] handling exit code: 0 2024-12-17 22:08:29,781 INFO SenderThread:81075 [sender.py:send_exit():588] handling runtime: 298 2024-12-17 22:08:29,782 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:08:29,782 INFO SenderThread:81075 [sender.py:send_exit():594] send defer 2024-12-17 22:08:29,782 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:29,782 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 0 2024-12-17 22:08:29,782 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:29,782 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 0 2024-12-17 22:08:29,782 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 1 2024-12-17 22:08:29,782 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:29,782 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 1 2024-12-17 22:08:29,782 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:29,782 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 1 2024-12-17 22:08:29,783 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 2 2024-12-17 22:08:29,783 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:29,783 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 2 2024-12-17 22:08:29,783 INFO HandlerThread:81075 [system_monitor.py:finish():203] Stopping system monitor 2024-12-17 22:08:29,783 DEBUG SystemMonitor:81075 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2024-12-17 22:08:29,783 INFO HandlerThread:81075 [interfaces.py:finish():200] Joined cpu monitor 2024-12-17 22:08:29,783 DEBUG SystemMonitor:81075 [system_monitor.py:_start():183] Publishing last batch of metrics 2024-12-17 22:08:29,783 INFO HandlerThread:81075 [interfaces.py:finish():200] Joined disk monitor 2024-12-17 22:08:29,847 INFO HandlerThread:81075 [interfaces.py:finish():200] Joined gpu monitor 2024-12-17 22:08:29,847 INFO HandlerThread:81075 [interfaces.py:finish():200] Joined memory monitor 2024-12-17 22:08:29,848 INFO HandlerThread:81075 [interfaces.py:finish():200] Joined network monitor 2024-12-17 22:08:29,848 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:29,848 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 2 2024-12-17 22:08:29,848 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 3 2024-12-17 22:08:29,848 DEBUG SenderThread:81075 [sender.py:send():379] send: stats 2024-12-17 22:08:29,848 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:29,848 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 3 2024-12-17 22:08:29,848 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:29,848 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 3 2024-12-17 22:08:29,849 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 4 2024-12-17 22:08:29,849 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:29,849 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 4 2024-12-17 22:08:30,222 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:08:30,222 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:08:30,222 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:30,458 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /tmp/tmp1icd4ri1wandb/dz9qfb6u-events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:08:35,850 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:35,851 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:35,851 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:35,851 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: keepalive 2024-12-17 22:08:35,851 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 4 2024-12-17 22:08:35,851 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 5 2024-12-17 22:08:35,851 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:08:35,851 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:35,852 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:08:35,852 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 5 2024-12-17 22:08:35,852 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:08:35,852 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:35,852 DEBUG SenderThread:81075 [sender.py:send():379] send: history 2024-12-17 22:08:35,853 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: summary_record 2024-12-17 22:08:35,856 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:08:35,856 DEBUG SenderThread:81075 [sender.py:send():379] send: summary 2024-12-17 22:08:35,857 INFO SenderThread:81075 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2024-12-17 22:08:35,857 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:35,857 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 5 2024-12-17 22:08:35,857 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 6 2024-12-17 22:08:35,857 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:35,857 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 6 2024-12-17 22:08:35,857 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:35,857 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 6 2024-12-17 22:08:35,857 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 7 2024-12-17 22:08:35,858 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: status_report 2024-12-17 22:08:35,858 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:35,858 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 7 2024-12-17 22:08:35,858 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:35,858 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 7 2024-12-17 22:08:36,224 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:08:36,782 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:37,389 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 8 2024-12-17 22:08:37,389 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:37,389 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:37,389 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 8 2024-12-17 22:08:37,390 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:37,390 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 8 2024-12-17 22:08:37,390 INFO SenderThread:81075 [job_builder.py:build():440] Attempting to build job artifact 2024-12-17 22:08:37,392 INFO SenderThread:81075 [job_builder.py:_get_source_type():569] is repo sourced job 2024-12-17 22:08:37,406 INFO SenderThread:81075 [job_builder.py:build():545] adding wandb-job metadata file 2024-12-17 22:08:37,420 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 9 2024-12-17 22:08:37,421 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:37,421 DEBUG SenderThread:81075 [sender.py:send():379] send: artifact 2024-12-17 22:08:37,421 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 9 2024-12-17 22:08:37,783 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:38,225 INFO wandb-upload_0:81075 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1kt9lw7n 2024-12-17 22:08:38,225 INFO wandb-upload_1:81075 [upload_job.py:push():85] Skipped uploading /tmp/tmp10rvpnms/wandb-job.json 2024-12-17 22:08:38,227 INFO Thread-12 :81075 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:38,334 INFO wandb-upload_2:81075 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpx66qg67q 2024-12-17 22:08:38,848 INFO SenderThread:81075 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzM3MTUyNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzM1Mzg0NQ==', 'versionIndex': 381}}} 2024-12-17 22:08:38,848 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:38,848 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 9 2024-12-17 22:08:38,848 INFO SenderThread:81075 [dir_watcher.py:finish():358] shutting down directory watcher 2024-12-17 22:08:39,228 INFO SenderThread:81075 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files 2024-12-17 22:08:39,228 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/config.yaml config.yaml 2024-12-17 22:08:39,228 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json wandb-summary.json 2024-12-17 22:08:39,228 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 events.out.tfevents.1734491011.leonmkim-ROG-Strix-G15CS-G15CS.80866.0 2024-12-17 22:08:39,231 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/requirements.txt requirements.txt 2024-12-17 22:08:39,231 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-metadata.json wandb-metadata.json 2024-12-17 22:08:39,231 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log output.log 2024-12-17 22:08:39,232 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/diff.patch diff.patch 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/code/FISH/eval_robot.py code/FISH/eval_robot.py 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 media/videos/eval/4_eval_4_f4b58612f8b6969704bc.mp4 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 media/videos/eval/0_eval_0_586a5a9c3ae7624982c1.mp4 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 media/videos/eval/3_eval_3_4c5ec7ff909b864f01e9.mp4 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 media/videos/eval/2_eval_2_f3929c65c66482ea37f4.mp4 2024-12-17 22:08:39,234 INFO SenderThread:81075 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 media/videos/eval/1_eval_1_80e3114e01360495cabc.mp4 2024-12-17 22:08:39,234 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 10 2024-12-17 22:08:39,234 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:39,235 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:39,235 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 10 2024-12-17 22:08:39,235 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:39,235 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 10 2024-12-17 22:08:39,235 INFO SenderThread:81075 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 22:08:39,411 INFO wandb-upload_2:81075 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/requirements.txt 2024-12-17 22:08:39,430 INFO wandb-upload_0:81075 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/config.yaml 2024-12-17 22:08:39,446 INFO wandb-upload_1:81075 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/wandb-summary.json 2024-12-17 22:08:39,460 INFO wandb-upload_3:81075 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/files/output.log 2024-12-17 22:08:39,660 INFO Thread-11 (_thread_body):81075 [sender.py:transition_state():614] send defer: 11 2024-12-17 22:08:39,660 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:39,660 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 11 2024-12-17 22:08:39,660 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:39,661 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 11 2024-12-17 22:08:39,661 INFO SenderThread:81075 [file_pusher.py:join():175] waiting for file pusher 2024-12-17 22:08:39,661 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 12 2024-12-17 22:08:39,662 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:39,662 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 12 2024-12-17 22:08:39,662 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:39,662 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 12 2024-12-17 22:08:39,662 INFO SenderThread:81075 [file_stream.py:finish():601] file stream finish called 2024-12-17 22:08:39,784 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:39,912 INFO SenderThread:81075 [file_stream.py:finish():605] file stream finish is done 2024-12-17 22:08:39,912 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 13 2024-12-17 22:08:39,912 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:39,912 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:39,912 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 13 2024-12-17 22:08:39,913 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:39,913 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 13 2024-12-17 22:08:39,913 INFO SenderThread:81075 [sender.py:transition_state():614] send defer: 14 2024-12-17 22:08:39,913 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: defer 2024-12-17 22:08:39,913 DEBUG SenderThread:81075 [sender.py:send():379] send: final 2024-12-17 22:08:39,913 INFO HandlerThread:81075 [handler.py:handle_request_defer():184] handle defer: 14 2024-12-17 22:08:39,913 DEBUG SenderThread:81075 [sender.py:send():379] send: footer 2024-12-17 22:08:39,913 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: defer 2024-12-17 22:08:39,913 INFO SenderThread:81075 [sender.py:send_request_defer():610] handle sender defer: 14 2024-12-17 22:08:39,914 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:39,914 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:39,914 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: server_info 2024-12-17 22:08:39,914 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: server_info 2024-12-17 22:08:39,914 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: poll_exit 2024-12-17 22:08:39,915 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: get_summary 2024-12-17 22:08:39,916 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: sampled_history 2024-12-17 22:08:39,916 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: internal_messages 2024-12-17 22:08:39,968 DEBUG SenderThread:81075 [sender.py:send_request():406] send_request: poll_exit 2024-12-17 22:08:39,968 INFO MainThread:81075 [wandb_run.py:_footer_history_summary_info():4016] rendering history 2024-12-17 22:08:39,968 INFO MainThread:81075 [wandb_run.py:_footer_history_summary_info():4048] rendering summary 2024-12-17 22:08:39,968 INFO MainThread:81075 [wandb_run.py:_footer_sync_info():3975] logging synced files 2024-12-17 22:08:39,969 DEBUG HandlerThread:81075 [handler.py:handle_request():158] handle_request: shutdown 2024-12-17 22:08:39,969 INFO HandlerThread:81075 [handler.py:finish():882] shutting down handler 2024-12-17 22:08:40,916 INFO WriterThread:81075 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/524_0/220324/wandb/run-20241217_220330-o350ecc4/run-o350ecc4.wandb 2024-12-17 22:08:40,968 INFO SenderThread:81075 [sender.py:finish():1615] shutting down sender 2024-12-17 22:08:40,968 INFO SenderThread:81075 [file_pusher.py:finish():169] shutting down file pusher 2024-12-17 22:08:40,968 INFO SenderThread:81075 [file_pusher.py:join():175] waiting for file pusher