2024-12-31 21:35:03,442 INFO    StreamThr :1748203 [internal.py:wandb_internal():85] W&B internal server running at pid: 1748203, started at: 2024-12-31 21:35:03.441695
2024-12-31 21:35:03,443 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status
2024-12-31 21:35:03,453 INFO    WriterThread:1748203 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/run-c7zbixne.wandb
2024-12-31 21:35:03,454 DEBUG   SenderThread:1748203 [sender.py:send():379] send: header
2024-12-31 21:35:03,464 DEBUG   SenderThread:1748203 [sender.py:send():379] send: run
2024-12-31 21:35:03,684 INFO    SenderThread:1748203 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files
2024-12-31 21:35:03,684 INFO    SenderThread:1748203 [sender.py:_start_run_threads():1188] run started: c7zbixne with start time 1735698903.442219
2024-12-31 21:35:03,701 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: check_version
2024-12-31 21:35:03,701 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: check_version
2024-12-31 21:35:03,776 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: run_start
2024-12-31 21:35:03,785 DEBUG   HandlerThread:1748203 [system_info.py:__init__():26] System info init
2024-12-31 21:35:03,785 DEBUG   HandlerThread:1748203 [system_info.py:__init__():41] System info init done
2024-12-31 21:35:03,785 INFO    HandlerThread:1748203 [system_monitor.py:start():194] Starting system monitor
2024-12-31 21:35:03,785 INFO    SystemMonitor:1748203 [system_monitor.py:_start():158] Starting system asset monitoring threads
2024-12-31 21:35:03,785 INFO    HandlerThread:1748203 [system_monitor.py:probe():214] Collecting system info
2024-12-31 21:35:03,785 INFO    SystemMonitor:1748203 [interfaces.py:start():188] Started cpu monitoring
2024-12-31 21:35:03,786 INFO    SystemMonitor:1748203 [interfaces.py:start():188] Started disk monitoring
2024-12-31 21:35:03,787 INFO    SystemMonitor:1748203 [interfaces.py:start():188] Started gpu monitoring
2024-12-31 21:35:03,788 INFO    SystemMonitor:1748203 [interfaces.py:start():188] Started memory monitoring
2024-12-31 21:35:03,788 INFO    SystemMonitor:1748203 [interfaces.py:start():188] Started network monitoring
2024-12-31 21:35:03,799 DEBUG   HandlerThread:1748203 [system_info.py:probe():152] Probing system
2024-12-31 21:35:03,800 DEBUG   HandlerThread:1748203 [system_info.py:_probe_git():137] Probing git
2024-12-31 21:35:03,805 DEBUG   HandlerThread:1748203 [system_info.py:_probe_git():145] Probing git done
2024-12-31 21:35:03,805 DEBUG   HandlerThread:1748203 [system_info.py:probe():200] Probing system done
2024-12-31 21:35:03,805 DEBUG   HandlerThread:1748203 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T02:35:03.799114', 'startedAt': '2025-01-01T02:35:03.434313', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.55275, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.978, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 515.321044921875}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2024-12-31 21:35:03,805 INFO    HandlerThread:1748203 [system_monitor.py:probe():224] Finished collecting system info
2024-12-31 21:35:03,805 INFO    HandlerThread:1748203 [system_monitor.py:probe():227] Publishing system info
2024-12-31 21:35:03,805 DEBUG   HandlerThread:1748203 [system_info.py:_save_code():44] Saving code
2024-12-31 21:35:03,806 DEBUG   HandlerThread:1748203 [system_info.py:_save_code():67] Saving code done
2024-12-31 21:35:03,806 DEBUG   HandlerThread:1748203 [system_info.py:_save_patches():84] Saving git patches
2024-12-31 21:35:03,868 DEBUG   HandlerThread:1748203 [system_info.py:_save_patches():126] Saving git patches done
2024-12-31 21:35:03,869 INFO    HandlerThread:1748203 [system_monitor.py:probe():229] Finished publishing system info
2024-12-31 21:35:03,872 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:35:03,872 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2024-12-31 21:35:03,872 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2024-12-31 21:35:03,872 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file diff.patch with policy now
2024-12-31 21:35:04,121 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: python_packages
2024-12-31 21:35:04,121 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: python_packages
2024-12-31 21:35:04,121 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:35:04,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:35:04,124 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/fcqxte4d-diff.patch
2024-12-31 21:35:04,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:04,125 INFO    HandlerThread:1748203 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/tb"
  save: true
}
control {
  relay_id: "0766f48012f149d2be96e80bee70b554"
}
_info {
  stream_id: "c7zbixne"
}

2024-12-31 21:35:04,126 DEBUG   HandlerThread:1748203 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2024-12-31 21:35:04,159 INFO    wandb-upload_0:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/xj2oosdr-wandb-metadata.json
2024-12-31 21:35:04,187 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/qe78rsai-code/FISH/eval_robot.py
2024-12-31 21:35:04,284 DEBUG   SenderThread:1748203 [sender.py:send():379] send: telemetry
2024-12-31 21:35:04,284 DEBUG   SenderThread:1748203 [sender.py:send():379] send: tbrecord
2024-12-31 21:35:04,285 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:35:04,285 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0 with policy live
2024-12-31 21:35:04,519 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/jafedgkg-events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:35:04,685 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-metadata.json
2024-12-31 21:35:04,685 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/diff.patch
2024-12-31 21:35:04,685 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/code/FISH/eval_robot.py
2024-12-31 21:35:04,686 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:35:04,686 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/requirements.txt
2024-12-31 21:35:04,686 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:04,686 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/code/FISH
2024-12-31 21:35:04,686 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/code
2024-12-31 21:35:05,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:06,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:06,684 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:07,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:08,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:09,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:09,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:10,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:11,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:12,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:13,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:14,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:14,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:15,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:16,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:17,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:18,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:18,759 DEBUG   SenderThread:1748203 [sender.py:send():379] send: config
2024-12-31 21:35:19,121 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:35:19,121 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:35:19,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:19,274 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:20,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:20,689 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:21,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:22,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:22,690 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:23,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:24,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:24,690 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:25,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:25,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:26,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:27,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:28,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:28,691 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:29,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:30,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:30,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:31,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:32,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:32,692 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:33,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:34,121 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:35:34,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:35:34,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:35,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:35,331 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:35,693 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/config.yaml
2024-12-31 21:35:36,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:36,694 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:37,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:38,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:38,694 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:39,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:40,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:40,451 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:40,695 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:41,127 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:42,127 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:42,695 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:43,127 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:44,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:44,696 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:45,127 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:45,520 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:46,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:46,697 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:47,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:48,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:48,697 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:49,121 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:35:49,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:35:49,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:50,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:50,639 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:50,698 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:51,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:52,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:52,698 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:53,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:54,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:54,699 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:55,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:55,841 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:35:56,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:56,700 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:57,129 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:58,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:35:58,700 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:35:59,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:00,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:00,701 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:01,117 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:01,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:02,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:02,701 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:03,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:03,788 DEBUG   SystemMonitor:1748203 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2024-12-31 21:36:03,789 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:36:04,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:36:04,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:36:04,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:04,702 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:05,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:06,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:06,354 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:06,702 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:07,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:08,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:08,703 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:09,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:10,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:10,372 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:36:10,372 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4 with policy now
2024-12-31 21:36:10,372 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:36:10,373 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:36:10,373 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:36:10,374 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:36:10,588 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/ii2pmaby-media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media
2024-12-31 21:36:10,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos
2024-12-31 21:36:10,961 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/t9ibgwop-media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4
2024-12-31 21:36:11,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:11,389 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:12,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:12,704 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:13,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:14,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:15,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:16,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:16,705 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:16,706 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:17,132 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:18,132 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:18,706 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:19,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:36:19,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:36:19,165 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:20,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:21,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:22,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:22,206 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:23,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:24,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:25,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:26,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:27,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:28,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:28,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:29,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:30,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:31,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:32,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:32,711 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:36:32,947 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/6vmh7e8o-events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:36:33,134 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:33,789 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:36:33,790 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:34,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:36:34,122 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:36:34,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:34,712 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:35,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:36,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:36,712 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:37,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:38,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:38,713 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:38,987 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:39,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:40,135 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:40,714 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:41,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:42,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:42,715 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:43,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:44,124 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:44,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:44,716 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:45,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:46,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:46,716 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:47,136 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:48,137 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:48,717 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:49,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:36:49,126 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:36:49,137 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:49,248 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:49,970 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:36:49,970 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4 with policy now
2024-12-31 21:36:49,972 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:36:49,973 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:36:49,973 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:36:49,973 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:36:50,137 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:50,222 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/8z93u7al-media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4
2024-12-31 21:36:50,718 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:50,718 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:36:50,718 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4
2024-12-31 21:36:50,718 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval
2024-12-31 21:36:50,978 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/8lbc22xp-media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4
2024-12-31 21:36:51,141 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:52,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:52,718 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:53,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:54,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:54,719 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:55,127 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:36:55,145 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:56,138 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:57,139 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:58,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:36:58,720 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:36:59,145 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:00,145 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:01,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:01,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:02,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:03,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:03,790 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:37:04,122 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:37:04,123 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:37:04,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:05,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:06,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:06,236 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:07,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:08,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:09,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:10,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:11,143 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:12,126 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:12,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:13,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:14,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:15,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:16,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:17,142 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:17,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:17,578 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:37:17,578 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:37:17,579 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:37:17,727 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:37:17,727 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:37:18,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:18,727 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:19,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:37:19,123 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:37:19,165 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:20,145 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:20,728 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:21,145 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:22,150 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:22,302 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:22,728 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:23,149 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:24,150 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:24,729 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:25,151 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:26,151 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:26,730 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:27,151 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:27,478 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:28,151 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:28,731 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:29,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:30,151 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:30,731 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:31,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:32,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:32,655 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:32,732 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:33,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:33,790 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:37:34,123 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:37:34,125 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:37:34,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:34,733 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:35,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:36,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:36,734 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:37,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:37,856 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:38,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:38,734 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:39,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:40,152 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:40,735 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:41,153 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:42,154 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:42,736 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:43,078 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:43,153 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:44,177 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:44,737 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:45,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:45,908 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:37:45,908 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4 with policy now
2024-12-31 21:37:45,913 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:37:45,913 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:37:45,913 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:37:45,914 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:37:46,119 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/ju9rtktg-media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4
2024-12-31 21:37:46,153 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:46,737 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:46,738 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:37:46,738 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4
2024-12-31 21:37:46,738 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval
2024-12-31 21:37:47,001 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/fvizvw0x-media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4
2024-12-31 21:37:47,158 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:48,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:48,738 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:48,945 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:49,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:37:49,125 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:37:49,165 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:50,157 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:50,739 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:51,157 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:52,156 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:52,739 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:37:53,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:54,130 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:54,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:55,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:56,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:57,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:58,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:37:59,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:37:59,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:00,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:01,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:02,162 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:03,163 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:03,791 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:38:04,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:38:04,125 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:38:04,161 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:04,290 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:04,414 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:38:04,414 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:38:04,414 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:38:04,743 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:38:04,744 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:04,744 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:38:04,991 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/abujme1x-events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:38:05,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:06,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:06,744 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:07,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:08,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:08,745 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:09,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:09,336 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:10,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:10,745 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:11,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:12,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:12,746 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:13,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:14,166 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:14,585 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:14,747 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:15,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:16,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:16,748 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:17,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:18,167 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:18,748 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:19,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:38:19,125 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:38:19,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:19,759 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:20,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:20,749 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:21,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:22,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:22,750 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:23,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:24,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:24,752 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:25,004 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:25,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:26,168 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:26,752 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:27,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:28,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:28,753 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:29,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:30,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:30,210 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:30,754 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:31,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:32,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:32,755 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:33,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:33,791 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:38:34,125 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:38:34,125 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:38:34,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:34,756 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:35,169 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:35,460 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:36,170 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:36,756 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:37,170 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:38,170 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:38,757 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:39,170 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:40,170 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:40,684 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:40,758 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:41,171 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:42,171 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:42,758 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:43,175 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:43,962 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:38:43,962 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4 with policy now
2024-12-31 21:38:43,970 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:38:43,971 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:38:43,971 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:38:43,978 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:38:44,172 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:44,184 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/2up8vcgt-media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4
2024-12-31 21:38:44,759 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:44,759 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:38:44,759 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4
2024-12-31 21:38:44,760 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval
2024-12-31 21:38:45,051 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/f95qu0cn-media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4
2024-12-31 21:38:45,177 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:46,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:46,182 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:47,187 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:48,179 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:48,761 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:49,132 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:38:49,132 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:38:49,188 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:50,183 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:51,183 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:51,465 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:52,193 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:52,762 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:38:53,185 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:54,194 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:55,192 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:56,193 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:57,139 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:38:57,187 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:58,195 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:38:59,199 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:00,195 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:01,200 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:02,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:03,138 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:03,196 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:03,792 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:39:04,133 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:39:04,133 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:39:04,195 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:05,201 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:06,197 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:06,487 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:39:06,487 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:39:06,487 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:39:06,767 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:39:06,767 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:06,767 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:39:07,019 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/4bycnqb2-events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:39:07,202 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:08,165 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:08,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:08,767 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:09,202 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:10,202 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:10,768 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:11,202 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:12,203 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:12,769 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:13,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:13,337 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:14,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:14,770 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:15,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:16,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:16,770 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:17,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:18,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:18,526 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:18,771 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:19,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:39:19,128 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:39:19,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:20,204 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:20,772 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:21,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:22,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:22,773 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:23,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:23,528 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:24,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:24,773 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:25,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:26,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:26,774 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:27,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:28,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:28,722 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:28,775 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:29,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:30,205 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:30,776 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:31,206 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:32,206 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:32,776 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:33,207 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:33,792 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:39:33,792 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:34,128 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:39:34,128 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:39:34,207 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:34,777 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:35,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:36,207 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:36,778 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:37,207 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:38,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:38,779 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:38,897 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:39,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:40,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:40,779 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:41,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:42,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:42,780 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:43,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:44,094 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:44,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:44,781 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:45,208 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:46,217 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:46,326 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:39:46,326 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4 with policy now
2024-12-31 21:39:46,345 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:39:46,346 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:39:46,346 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:39:46,364 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:39:46,546 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/gxxmyc6m-media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4
2024-12-31 21:39:46,782 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:46,782 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:39:46,782 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4
2024-12-31 21:39:46,782 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval
2024-12-31 21:39:47,046 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/wr4ci5gj-media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4
2024-12-31 21:39:47,239 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:48,246 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:48,782 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:49,132 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:39:49,132 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:39:49,234 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:49,287 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:50,237 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:50,783 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:51,241 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:52,241 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:53,241 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:54,243 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:54,785 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:39:55,148 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:39:55,270 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:56,253 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:57,250 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:58,252 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:39:59,257 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:00,251 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:01,144 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:40:01,260 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:02,252 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:03,262 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:03,792 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:40:04,131 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: stop_status
2024-12-31 21:40:04,132 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: stop_status
2024-12-31 21:40:04,259 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:04,788 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:40:05,165 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: log_artifact
2024-12-31 21:40:05,166 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: log_artifact
2024-12-31 21:40:05,262 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:05,431 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:40:05,626 INFO    wandb-upload_2:1748203 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp8715f46y
2024-12-31 21:40:05,784 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: log_artifact
2024-12-31 21:40:05,788 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:40:05,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json
2024-12-31 21:40:05,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json
2024-12-31 21:40:05,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media
2024-12-31 21:40:05,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table/eval
2024-12-31 21:40:05,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table
2024-12-31 21:40:05,795 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: partial_history
2024-12-31 21:40:06,026 INFO    SenderThread:1748203 [sender.py:send_request_log_artifact():1525] logged artifact run-c7zbixne-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDEzMzI0NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2OTAxMg==', 'latestArtifact': None}}
2024-12-31 21:40:06,026 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:40:06,026 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json with policy now
2024-12-31 21:40:06,027 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:40:06,028 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,029 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,029 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:40:06,029 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,030 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,030 DEBUG   SenderThread:1748203 [sender.py:send():379] send: telemetry
2024-12-31 21:40:06,031 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,031 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,032 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,032 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,033 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,033 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,034 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,034 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,034 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,035 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,035 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,035 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,035 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,036 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,036 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,036 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,036 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,037 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,037 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,037 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,037 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,038 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:06,038 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: log_artifact
2024-12-31 21:40:06,219 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/m7nfgzio-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json
2024-12-31 21:40:06,243 INFO    wandb-upload_0:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/mu41qi4m-media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json
2024-12-31 21:40:06,255 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:06,459 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/hh7yudnl-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json
2024-12-31 21:40:06,506 INFO    wandb-upload_1:1748203 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpv5ousy4x
2024-12-31 21:40:06,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:40:06,874 INFO    SenderThread:1748203 [sender.py:send_request_log_artifact():1525] logged artifact run-c7zbixne-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDEzMzI2Mw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2OTAxNw==', 'latestArtifact': None}}
2024-12-31 21:40:06,874 DEBUG   SenderThread:1748203 [sender.py:send():379] send: files
2024-12-31 21:40:06,874 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json with policy now
2024-12-31 21:40:06,875 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:40:06,875 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:06,875 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:40:06,876 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:07,117 INFO    wandb-upload_0:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/cy3wmssk-media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json
2024-12-31 21:40:07,255 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:07,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:40:07,789 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:40:08,255 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:09,255 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:09,501 DEBUG   SenderThread:1748203 [sender.py:send():379] send: exit
2024-12-31 21:40:09,501 INFO    SenderThread:1748203 [sender.py:send_exit():586] handling exit code: 0
2024-12-31 21:40:09,502 INFO    SenderThread:1748203 [sender.py:send_exit():588] handling runtime: 305
2024-12-31 21:40:09,502 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:09,502 INFO    SenderThread:1748203 [sender.py:send_exit():594] send defer
2024-12-31 21:40:09,502 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:09,502 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 0
2024-12-31 21:40:09,502 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:09,502 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 0
2024-12-31 21:40:09,502 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 1
2024-12-31 21:40:09,502 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:09,503 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 1
2024-12-31 21:40:09,503 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:09,503 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 1
2024-12-31 21:40:09,503 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 2
2024-12-31 21:40:09,503 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:09,503 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 2
2024-12-31 21:40:09,503 INFO    HandlerThread:1748203 [system_monitor.py:finish():203] Stopping system monitor
2024-12-31 21:40:09,503 DEBUG   SystemMonitor:1748203 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2024-12-31 21:40:09,503 DEBUG   SystemMonitor:1748203 [system_monitor.py:_start():183] Publishing last batch of metrics
2024-12-31 21:40:09,504 INFO    HandlerThread:1748203 [interfaces.py:finish():200] Joined cpu monitor
2024-12-31 21:40:09,504 INFO    HandlerThread:1748203 [interfaces.py:finish():200] Joined disk monitor
2024-12-31 21:40:09,556 INFO    HandlerThread:1748203 [interfaces.py:finish():200] Joined gpu monitor
2024-12-31 21:40:09,556 INFO    HandlerThread:1748203 [interfaces.py:finish():200] Joined memory monitor
2024-12-31 21:40:09,556 INFO    HandlerThread:1748203 [interfaces.py:finish():200] Joined network monitor
2024-12-31 21:40:09,557 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:09,557 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 2
2024-12-31 21:40:09,557 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 3
2024-12-31 21:40:09,557 DEBUG   SenderThread:1748203 [sender.py:send():379] send: stats
2024-12-31 21:40:09,557 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:09,557 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 3
2024-12-31 21:40:09,557 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:09,557 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 3
2024-12-31 21:40:09,557 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 4
2024-12-31 21:40:09,557 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:09,557 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 4
2024-12-31 21:40:09,790 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:40:11,791 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:40:15,561 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:15,561 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:40:15,561 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: keepalive
2024-12-31 21:40:15,562 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:15,562 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 4
2024-12-31 21:40:15,562 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 5
2024-12-31 21:40:15,563 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:15,563 DEBUG   SenderThread:1748203 [sender.py:send():379] send: history
2024-12-31 21:40:15,563 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: summary_record
2024-12-31 21:40:15,563 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:15,563 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 5
2024-12-31 21:40:15,564 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:15,564 DEBUG   SenderThread:1748203 [sender.py:send():379] send: summary
2024-12-31 21:40:15,568 INFO    SenderThread:1748203 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2024-12-31 21:40:15,568 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:15,568 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 5
2024-12-31 21:40:15,568 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 6
2024-12-31 21:40:15,568 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:15,568 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 6
2024-12-31 21:40:15,568 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:15,568 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 6
2024-12-31 21:40:15,572 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: status_report
2024-12-31 21:40:15,824 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 7
2024-12-31 21:40:15,824 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:15,825 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 7
2024-12-31 21:40:15,825 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:15,825 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 7
2024-12-31 21:40:15,825 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:40:16,502 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:16,825 INFO    Thread-12 :1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/config.yaml
2024-12-31 21:40:18,018 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 8
2024-12-31 21:40:18,019 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:18,019 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:18,019 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 8
2024-12-31 21:40:18,019 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:18,019 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 8
2024-12-31 21:40:18,019 INFO    SenderThread:1748203 [job_builder.py:build():440] Attempting to build job artifact
2024-12-31 21:40:18,021 INFO    SenderThread:1748203 [job_builder.py:_get_source_type():569] is repo sourced job
2024-12-31 21:40:18,023 INFO    SenderThread:1748203 [job_builder.py:build():545] adding wandb-job metadata file
2024-12-31 21:40:18,031 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 9
2024-12-31 21:40:18,031 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:18,032 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 9
2024-12-31 21:40:18,032 DEBUG   SenderThread:1748203 [sender.py:send():379] send: artifact
2024-12-31 21:40:18,448 INFO    SenderThread:1748203 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDEzMDIwNA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDEzMDIwNA==', 'versionIndex': 393}}}
2024-12-31 21:40:18,448 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:18,448 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 9
2024-12-31 21:40:18,448 INFO    SenderThread:1748203 [dir_watcher.py:finish():358] shutting down directory watcher
2024-12-31 21:40:18,503 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/config.yaml config.yaml
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0 events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json wandb-summary.json
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/requirements.txt requirements.txt
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-metadata.json wandb-metadata.json
2024-12-31 21:40:18,827 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log output.log
2024-12-31 21:40:18,828 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/diff.patch diff.patch
2024-12-31 21:40:18,828 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2024-12-31 21:40:18,829 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4 media/videos/eval/4_eval_4_578f39c6eeca4936625c.mp4
2024-12-31 21:40:18,830 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4 media/videos/eval/1_eval_1_eb902dfcfac85de57271.mp4
2024-12-31 21:40:18,830 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4 media/videos/eval/3_eval_3_d10ed98fe5d597fa5a89.mp4
2024-12-31 21:40:18,830 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4 media/videos/eval/2_eval_2_dea02de20bdd586cd5fd.mp4
2024-12-31 21:40:18,830 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4 media/videos/eval/0_eval_0_c8a408e3df7bfc74cc89.mp4
2024-12-31 21:40:18,834 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json media/table/eval/total_success_rate_ci_6_03024be658973921c64a.table.json
2024-12-31 21:40:18,835 INFO    SenderThread:1748203 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json
2024-12-31 21:40:18,835 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 10
2024-12-31 21:40:18,835 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:18,835 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:18,835 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 10
2024-12-31 21:40:18,837 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:18,837 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 10
2024-12-31 21:40:18,837 INFO    SenderThread:1748203 [file_pusher.py:finish():169] shutting down file pusher
2024-12-31 21:40:19,007 INFO    wandb-upload_0:1748203 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/wandb-summary.json
2024-12-31 21:40:19,008 INFO    wandb-upload_1:1748203 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/config.yaml
2024-12-31 21:40:19,064 INFO    wandb-upload_2:1748203 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/requirements.txt
2024-12-31 21:40:19,074 INFO    wandb-upload_4:1748203 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/files/output.log
2024-12-31 21:40:19,110 INFO    wandb-upload_3:1748203 [upload_job.py:push():130] Uploaded file /tmp/tmp2ssyz0liwandb/edl5p080-events.out.tfevents.1735698904.leonmkim-ROG-Strix-G15CS-G15CS.1748029.0
2024-12-31 21:40:19,310 INFO    Thread-11 (_thread_body):1748203 [sender.py:transition_state():614] send defer: 11
2024-12-31 21:40:19,311 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:19,311 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 11
2024-12-31 21:40:19,311 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:19,311 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 11
2024-12-31 21:40:19,311 INFO    SenderThread:1748203 [file_pusher.py:join():175] waiting for file pusher
2024-12-31 21:40:19,314 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 12
2024-12-31 21:40:19,314 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:19,314 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 12
2024-12-31 21:40:19,314 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:19,314 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 12
2024-12-31 21:40:19,315 INFO    SenderThread:1748203 [file_stream.py:finish():601] file stream finish called
2024-12-31 21:40:19,504 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:19,533 INFO    SenderThread:1748203 [file_stream.py:finish():605] file stream finish is done
2024-12-31 21:40:19,533 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 13
2024-12-31 21:40:19,533 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:19,533 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:19,533 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 13
2024-12-31 21:40:19,533 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:19,533 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 13
2024-12-31 21:40:19,534 INFO    SenderThread:1748203 [sender.py:transition_state():614] send defer: 14
2024-12-31 21:40:19,534 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: defer
2024-12-31 21:40:19,534 INFO    HandlerThread:1748203 [handler.py:handle_request_defer():184] handle defer: 14
2024-12-31 21:40:19,534 DEBUG   SenderThread:1748203 [sender.py:send():379] send: final
2024-12-31 21:40:19,534 DEBUG   SenderThread:1748203 [sender.py:send():379] send: footer
2024-12-31 21:40:19,535 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: defer
2024-12-31 21:40:19,535 INFO    SenderThread:1748203 [sender.py:send_request_defer():610] handle sender defer: 14
2024-12-31 21:40:19,535 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:19,535 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: server_info
2024-12-31 21:40:19,535 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: poll_exit
2024-12-31 21:40:19,535 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: get_summary
2024-12-31 21:40:19,536 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: sampled_history
2024-12-31 21:40:19,536 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: internal_messages
2024-12-31 21:40:19,537 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:19,537 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: server_info
2024-12-31 21:40:19,603 DEBUG   SenderThread:1748203 [sender.py:send_request():406] send_request: poll_exit
2024-12-31 21:40:19,604 INFO    MainThread:1748203 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2024-12-31 21:40:19,604 INFO    MainThread:1748203 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2024-12-31 21:40:19,604 INFO    MainThread:1748203 [wandb_run.py:_footer_sync_info():3975] logging synced files
2024-12-31 21:40:19,604 DEBUG   HandlerThread:1748203 [handler.py:handle_request():158] handle_request: shutdown
2024-12-31 21:40:19,605 INFO    HandlerThread:1748203 [handler.py:finish():882] shutting down handler
2024-12-31 21:40:20,537 INFO    WriterThread:1748203 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/c7zbixne/wandb/run-20241231_213503-c7zbixne/run-c7zbixne.wandb
2024-12-31 21:40:20,604 INFO    SenderThread:1748203 [sender.py:finish():1615] shutting down sender
2024-12-31 21:40:20,604 INFO    SenderThread:1748203 [file_pusher.py:finish():169] shutting down file pusher
2024-12-31 21:40:20,604 INFO    SenderThread:1748203 [file_pusher.py:join():175] waiting for file pusher