2025-01-06 20:26:08,827 INFO    StreamThr :2109502 [internal.py:wandb_internal():85] W&B internal server running at pid: 2109502, started at: 2025-01-06 20:26:08.826881
2025-01-06 20:26:08,828 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status
2025-01-06 20:26:08,839 INFO    WriterThread:2109502 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/run-yki2vgk2.wandb
2025-01-06 20:26:08,839 DEBUG   SenderThread:2109502 [sender.py:send():379] send: header
2025-01-06 20:26:08,848 DEBUG   SenderThread:2109502 [sender.py:send():379] send: run
2025-01-06 20:26:09,131 INFO    SenderThread:2109502 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files
2025-01-06 20:26:09,131 INFO    SenderThread:2109502 [sender.py:_start_run_threads():1188] run started: yki2vgk2 with start time 1736213168.826926
2025-01-06 20:26:09,152 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: check_version
2025-01-06 20:26:09,152 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: check_version
2025-01-06 20:26:09,227 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: run_start
2025-01-06 20:26:09,232 DEBUG   HandlerThread:2109502 [system_info.py:__init__():26] System info init
2025-01-06 20:26:09,232 DEBUG   HandlerThread:2109502 [system_info.py:__init__():41] System info init done
2025-01-06 20:26:09,232 INFO    HandlerThread:2109502 [system_monitor.py:start():194] Starting system monitor
2025-01-06 20:26:09,232 INFO    SystemMonitor:2109502 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-06 20:26:09,232 INFO    HandlerThread:2109502 [system_monitor.py:probe():214] Collecting system info
2025-01-06 20:26:09,232 INFO    SystemMonitor:2109502 [interfaces.py:start():188] Started cpu monitoring
2025-01-06 20:26:09,233 INFO    SystemMonitor:2109502 [interfaces.py:start():188] Started disk monitoring
2025-01-06 20:26:09,233 INFO    SystemMonitor:2109502 [interfaces.py:start():188] Started gpu monitoring
2025-01-06 20:26:09,234 INFO    SystemMonitor:2109502 [interfaces.py:start():188] Started memory monitoring
2025-01-06 20:26:09,234 INFO    SystemMonitor:2109502 [interfaces.py:start():188] Started network monitoring
2025-01-06 20:26:09,247 DEBUG   HandlerThread:2109502 [system_info.py:probe():152] Probing system
2025-01-06 20:26:09,250 DEBUG   HandlerThread:2109502 [system_info.py:_probe_git():137] Probing git
2025-01-06 20:26:09,256 DEBUG   HandlerThread:2109502 [system_info.py:_probe_git():145] Probing git done
2025-01-06 20:26:09,256 DEBUG   HandlerThread:2109502 [system_info.py:probe():200] Probing system done
2025-01-06 20:26:09,256 DEBUG   HandlerThread:2109502 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T01:26:09.247912', 'startedAt': '2025-01-07T01:26:08.819097', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4123.881375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.983, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 486.81128311157227}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-06 20:26:09,256 INFO    HandlerThread:2109502 [system_monitor.py:probe():224] Finished collecting system info
2025-01-06 20:26:09,256 INFO    HandlerThread:2109502 [system_monitor.py:probe():227] Publishing system info
2025-01-06 20:26:09,256 DEBUG   HandlerThread:2109502 [system_info.py:_save_code():44] Saving code
2025-01-06 20:26:09,258 DEBUG   HandlerThread:2109502 [system_info.py:_save_code():67] Saving code done
2025-01-06 20:26:09,258 DEBUG   HandlerThread:2109502 [system_info.py:_save_patches():84] Saving git patches
2025-01-06 20:26:09,337 DEBUG   HandlerThread:2109502 [system_info.py:_save_patches():126] Saving git patches done
2025-01-06 20:26:09,337 INFO    HandlerThread:2109502 [system_monitor.py:probe():229] Finished publishing system info
2025-01-06 20:26:09,340 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:26:09,341 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-06 20:26:09,341 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-06 20:26:09,341 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-06 20:26:09,591 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: python_packages
2025-01-06 20:26:09,591 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: python_packages
2025-01-06 20:26:09,593 DEBUG   SenderThread:2109502 [sender.py:send():379] send: telemetry
2025-01-06 20:26:09,617 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/i4ptlrkj-diff.patch
2025-01-06 20:26:09,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:26:09,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:09,633 INFO    HandlerThread:2109502 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/tb"
  save: true
}
control {
  relay_id: "b301f1be74924b54993d6ce4bfe31f3d"
}
_info {
  stream_id: "yki2vgk2"
}

2025-01-06 20:26:09,633 DEBUG   HandlerThread:2109502 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-06 20:26:09,634 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:26:09,652 INFO    wandb-upload_1:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/ld6f0y1s-code/FISH/eval_robot.py
2025-01-06 20:26:09,660 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/89yc0fnz-wandb-metadata.json
2025-01-06 20:26:09,706 DEBUG   SenderThread:2109502 [sender.py:send():379] send: tbrecord
2025-01-06 20:26:09,707 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:26:09,707 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0 with policy live
2025-01-06 20:26:09,897 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/b5avmwwa-events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/requirements.txt
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/code/FISH/eval_robot.py
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-metadata.json
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/diff.patch
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/code
2025-01-06 20:26:10,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/code/FISH
2025-01-06 20:26:10,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:11,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:12,132 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:12,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:13,481 DEBUG   SenderThread:2109502 [sender.py:send():379] send: config
2025-01-06 20:26:13,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:14,489 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:14,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:15,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:16,133 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:16,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:17,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:18,134 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:18,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:19,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:19,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:20,134 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:20,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:21,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:22,135 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:22,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:23,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:24,591 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:26:24,592 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:26:24,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:24,755 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:25,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:26,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:27,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:28,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:29,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:30,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:30,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:31,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:32,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:33,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:34,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:35,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:35,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:36,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:37,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:38,594 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:39,591 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:26:39,592 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:26:39,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:40,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:40,660 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:41,140 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/config.yaml
2025-01-06 20:26:41,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:42,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:43,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:44,141 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:44,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:45,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:46,141 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:46,282 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:46,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:47,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:48,145 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:26:48,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:49,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:50,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:51,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:51,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:52,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:53,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:54,591 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:26:54,592 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:26:54,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:55,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:56,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:56,748 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:26:57,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:58,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:26:59,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:00,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:01,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:02,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:02,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:03,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:04,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:05,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:06,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:07,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:07,599 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:08,599 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:09,235 DEBUG   SystemMonitor:2109502 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-06 20:27:09,236 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:27:09,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:27:09,592 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:27:09,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:10,599 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:11,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:12,600 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:12,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:13,600 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:14,600 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:15,600 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:16,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:17,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:18,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:18,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:19,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:20,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:20,846 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:27:20,847 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4 with policy now
2025-01-06 20:27:20,847 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:27:20,848 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:27:20,848 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:27:20,849 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:27:21,115 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/jfkld2v2-media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4
2025-01-06 20:27:21,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4
2025-01-06 20:27:21,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:27:21,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval
2025-01-06 20:27:21,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media
2025-01-06 20:27:21,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos
2025-01-06 20:27:21,436 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/0dw3buau-media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4
2025-01-06 20:27:21,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:22,154 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:27:22,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:23,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:23,864 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:24,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:27:24,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:27:24,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:25,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:26,156 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:27:26,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:27,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:28,156 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:27:28,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:29,225 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:29,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:30,157 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:27:30,402 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/ob9gybci-events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:27:30,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:31,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:32,158 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:27:32,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:33,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:34,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:34,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:35,603 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:36,159 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:27:36,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:37,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:38,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:39,236 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:27:39,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:27:39,592 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:27:39,637 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:39,711 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:40,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:41,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:42,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:43,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:44,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:45,595 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:45,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:46,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:47,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:48,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:49,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:50,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:51,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:51,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:52,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:53,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:54,592 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:27:54,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:27:54,637 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:55,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:56,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:56,752 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:27:57,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:58,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:27:59,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:00,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:01,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:02,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:02,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:03,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:04,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:05,608 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:06,608 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:07,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:07,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:08,608 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:09,287 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:28:09,479 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:28:09,479 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4 with policy now
2025-01-06 20:28:09,482 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:28:09,482 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:28:09,483 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:28:09,492 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:28:09,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:28:09,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:28:09,608 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:09,752 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/z7yvm66i-media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4
2025-01-06 20:28:10,170 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:10,170 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:28:10,170 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4
2025-01-06 20:28:10,171 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval
2025-01-06 20:28:10,457 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/qbzvzbmx-media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4
2025-01-06 20:28:10,620 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:11,612 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:12,171 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:12,611 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:12,726 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:13,611 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:14,172 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:14,619 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:15,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:16,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:17,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:17,925 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:18,173 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:18,173 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:28:18,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:19,062 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:28:19,062 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:28:19,062 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:28:19,173 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:28:19,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:20,173 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:20,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:21,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:22,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:23,139 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:23,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:24,175 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:24,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:28:24,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:28:24,637 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:25,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:26,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:27,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:28,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:28,757 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:29,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:30,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:31,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:32,615 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:33,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:34,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:34,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:35,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:36,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:37,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:38,616 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:39,237 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:28:39,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:28:39,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:28:39,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:39,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:40,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:41,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:42,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:43,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:44,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:45,599 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:45,617 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:46,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:47,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:48,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:49,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:50,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:51,599 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:51,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:52,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:53,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:54,593 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:28:54,593 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:28:54,641 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:55,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:56,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:56,742 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:28:57,620 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:58,186 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:28:58,290 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:28:58,290 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4 with policy now
2025-01-06 20:28:58,295 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:28:58,296 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:28:58,296 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:28:58,296 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:28:58,571 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/wrpgfzrc-media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4
2025-01-06 20:28:58,619 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:28:59,186 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:28:59,187 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4
2025-01-06 20:28:59,187 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval
2025-01-06 20:28:59,474 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/sg62uxy3-media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4
2025-01-06 20:28:59,622 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:00,187 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:00,626 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:01,187 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:01,624 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:02,039 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:02,625 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:03,627 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:04,188 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:04,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:05,627 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:06,189 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:06,629 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:07,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:07,810 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:08,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:08,906 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:29:08,907 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:29:08,907 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:29:09,190 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:29:09,190 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:29:09,238 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:29:09,436 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/q5acwjzs-events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:29:09,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:29:09,597 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:29:09,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:10,190 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:10,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:11,634 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:12,629 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:13,495 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:13,629 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:14,192 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:14,629 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:15,629 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:16,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:17,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:18,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:18,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:19,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:20,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:21,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:22,630 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:23,601 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:23,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:24,596 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:29:24,596 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:29:24,637 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:25,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:26,196 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:26,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:27,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:28,196 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:28,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:29,250 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:29,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:30,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:31,631 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:32,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:33,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:34,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:34,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:35,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:36,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:37,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:38,632 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:39,238 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:29:39,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:29:39,597 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:29:39,641 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:39,761 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:40,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:41,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:42,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:43,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:44,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:45,602 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:45,633 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:46,634 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:47,636 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:48,096 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:29:48,096 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4 with policy now
2025-01-06 20:29:48,099 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:29:48,099 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:29:48,100 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:29:48,120 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:29:48,203 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:48,203 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:29:48,204 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4
2025-01-06 20:29:48,204 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval
2025-01-06 20:29:48,358 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/ek88t3dl-media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4
2025-01-06 20:29:48,637 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:48,649 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/1yvzy2cp-media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4
2025-01-06 20:29:49,639 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:50,204 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:50,604 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:50,641 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:51,639 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:52,642 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:53,645 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:54,205 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:54,621 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:29:54,621 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:29:54,639 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:55,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:55,714 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:29:56,206 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:29:56,644 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:57,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:57,971 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:29:57,971 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:29:57,972 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:29:58,207 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:29:58,207 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:29:58,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:29:59,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:00,207 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:00,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:01,606 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:01,685 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:02,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:03,645 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:04,209 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:04,645 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:05,645 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:06,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:07,202 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:07,646 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:08,210 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:08,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:09,238 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:30:09,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:30:09,597 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:30:09,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:10,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:11,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:12,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:12,739 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:13,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:14,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:15,647 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:16,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:17,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:18,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:18,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:19,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:20,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:21,648 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:22,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:23,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:24,597 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:30:24,597 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:30:24,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:24,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:25,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:26,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:27,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:28,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:29,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:30,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:30,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:31,649 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:32,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:33,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:34,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:35,605 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:35,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:36,650 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:37,653 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:37,904 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:30:37,904 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4 with policy now
2025-01-06 20:30:37,907 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:30:37,908 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:30:37,908 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:37,931 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:38,170 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/2vwrzyhy-media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4
2025-01-06 20:30:38,221 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:38,221 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:30:38,221 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4
2025-01-06 20:30:38,221 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval
2025-01-06 20:30:38,498 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/fu5tcay3-media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4
2025-01-06 20:30:38,658 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:39,239 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:30:39,600 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:30:39,601 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:30:39,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:40,221 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:40,654 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:40,765 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:41,663 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:42,661 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:43,662 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:44,223 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:44,660 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:45,661 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:46,223 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:46,614 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:46,664 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:47,224 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:30:47,477 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/89vdo28g-events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:30:47,490 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 20:30:47,491 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 20:30:47,618 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:30:47,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:47,977 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-06 20:30:47,986 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: partial_history
2025-01-06 20:30:48,055 INFO    wandb-upload_2:2109502 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphzpybba4
2025-01-06 20:30:48,224 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:48,224 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:30:48,224 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json
2025-01-06 20:30:48,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json
2025-01-06 20:30:48,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media
2025-01-06 20:30:48,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table
2025-01-06 20:30:48,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table/eval
2025-01-06 20:30:48,484 INFO    SenderThread:2109502 [sender.py:send_request_log_artifact():1525] logged artifact run-yki2vgk2-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjI3ODI5OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDYyODAyMg==', 'latestArtifact': None}}
2025-01-06 20:30:48,484 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:30:48,484 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now
2025-01-06 20:30:48,484 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:30:48,484 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,485 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,485 DEBUG   SenderThread:2109502 [sender.py:send():379] send: telemetry
2025-01-06 20:30:48,485 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,485 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,485 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,486 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,486 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,486 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,486 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,487 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,487 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,487 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,487 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,488 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,488 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,488 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,488 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,488 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,489 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,489 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,489 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,489 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,490 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:48,490 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:48,490 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: log_artifact
2025-01-06 20:30:48,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:48,671 INFO    wandb-upload_1:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/i46oru2s-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json
2025-01-06 20:30:48,712 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/f8uew4hb-media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json
2025-01-06 20:30:48,884 INFO    wandb-upload_2:2109502 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpjdukw6tv
2025-01-06 20:30:48,947 INFO    wandb-upload_1:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/isu3w40p-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json
2025-01-06 20:30:49,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:30:49,261 INFO    SenderThread:2109502 [sender.py:send_request_log_artifact():1525] logged artifact run-yki2vgk2-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjI3ODM3MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDYyODA4Mg==', 'latestArtifact': None}}
2025-01-06 20:30:49,261 DEBUG   SenderThread:2109502 [sender.py:send():379] send: files
2025-01-06 20:30:49,261 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json with policy now
2025-01-06 20:30:49,261 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:30:49,261 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:49,262 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:49,262 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:30:49,262 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:30:49,262 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:49,505 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/faz1rzt3-media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json
2025-01-06 20:30:49,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:50,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:30:50,225 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:30:50,656 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:51,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:52,607 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:30:52,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:53,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:54,598 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: stop_status
2025-01-06 20:30:54,598 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: stop_status
2025-01-06 20:30:54,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:55,657 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:30:55,937 DEBUG   SenderThread:2109502 [sender.py:send():379] send: exit
2025-01-06 20:30:55,937 INFO    SenderThread:2109502 [sender.py:send_exit():586] handling exit code: 0
2025-01-06 20:30:55,938 INFO    SenderThread:2109502 [sender.py:send_exit():588] handling runtime: 286
2025-01-06 20:30:55,938 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:30:55,938 INFO    SenderThread:2109502 [sender.py:send_exit():594] send defer
2025-01-06 20:30:55,938 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:30:55,938 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-06 20:30:55,938 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:30:55,938 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-06 20:30:55,938 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 1
2025-01-06 20:30:55,939 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:30:55,939 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-06 20:30:55,939 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:30:55,939 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-06 20:30:55,939 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 2
2025-01-06 20:30:55,939 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:30:55,939 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-06 20:30:55,939 INFO    HandlerThread:2109502 [system_monitor.py:finish():203] Stopping system monitor
2025-01-06 20:30:55,939 DEBUG   SystemMonitor:2109502 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-06 20:30:55,939 DEBUG   SystemMonitor:2109502 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-06 20:30:55,940 INFO    HandlerThread:2109502 [interfaces.py:finish():200] Joined cpu monitor
2025-01-06 20:30:55,940 INFO    HandlerThread:2109502 [interfaces.py:finish():200] Joined disk monitor
2025-01-06 20:30:56,005 INFO    HandlerThread:2109502 [interfaces.py:finish():200] Joined gpu monitor
2025-01-06 20:30:56,005 INFO    HandlerThread:2109502 [interfaces.py:finish():200] Joined memory monitor
2025-01-06 20:30:56,005 INFO    HandlerThread:2109502 [interfaces.py:finish():200] Joined network monitor
2025-01-06 20:30:56,005 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:30:56,005 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-06 20:30:56,005 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 3
2025-01-06 20:30:56,006 DEBUG   SenderThread:2109502 [sender.py:send():379] send: stats
2025-01-06 20:30:56,006 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:30:56,006 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-06 20:30:56,006 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:30:56,006 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-06 20:30:56,006 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 4
2025-01-06 20:30:56,006 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:30:56,007 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-06 20:30:56,227 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:30:58,228 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:31:02,038 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:02,038 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:02,038 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:31:02,038 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-06 20:31:02,038 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 5
2025-01-06 20:31:02,038 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: keepalive
2025-01-06 20:31:02,039 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:02,039 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:02,040 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-06 20:31:02,040 DEBUG   SenderThread:2109502 [sender.py:send():379] send: history
2025-01-06 20:31:02,041 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: summary_record
2025-01-06 20:31:02,041 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:31:02,042 DEBUG   SenderThread:2109502 [sender.py:send():379] send: summary
2025-01-06 20:31:02,043 INFO    SenderThread:2109502 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-06 20:31:02,043 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:02,043 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-06 20:31:02,043 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 6
2025-01-06 20:31:02,043 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:02,043 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-06 20:31:02,044 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:02,044 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-06 20:31:02,049 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: status_report
2025-01-06 20:31:02,184 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 7
2025-01-06 20:31:02,184 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:02,184 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-06 20:31:02,185 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:02,185 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-06 20:31:02,230 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/config.yaml
2025-01-06 20:31:02,230 INFO    Thread-12 :2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:31:02,939 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:04,333 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 8
2025-01-06 20:31:04,333 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:04,334 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:04,334 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-06 20:31:04,334 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:04,334 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-06 20:31:04,334 INFO    SenderThread:2109502 [job_builder.py:build():440] Attempting to build job artifact
2025-01-06 20:31:04,336 INFO    SenderThread:2109502 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-06 20:31:04,338 INFO    SenderThread:2109502 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-06 20:31:04,350 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 9
2025-01-06 20:31:04,350 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:04,350 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-06 20:31:04,351 DEBUG   SenderThread:2109502 [sender.py:send():379] send: artifact
2025-01-06 20:31:04,554 INFO    SenderThread:2109502 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTk1OTYxMQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}}
2025-01-06 20:31:04,554 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:04,554 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-06 20:31:04,554 INFO    SenderThread:2109502 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-06 20:31:04,939 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:05,231 INFO    SenderThread:2109502 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:31:05,231 INFO    SenderThread:2109502 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/config.yaml config.yaml
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json wandb-summary.json
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/requirements.txt requirements.txt
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-metadata.json wandb-metadata.json
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0 events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log output.log
2025-01-06 20:31:05,232 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/diff.patch diff.patch
2025-01-06 20:31:05,233 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-06 20:31:05,239 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4 media/videos/eval/0_eval_0_8bea6ae343656b6552f0.mp4
2025-01-06 20:31:05,239 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4 media/videos/eval/4_eval_4_31eece21c7aad1709607.mp4
2025-01-06 20:31:05,239 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4 media/videos/eval/1_eval_1_2d61b3be29847248470e.mp4
2025-01-06 20:31:05,239 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4 media/videos/eval/2_eval_2_f6a1e4b3986fcd7ef5c0.mp4
2025-01-06 20:31:05,239 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4 media/videos/eval/3_eval_3_561c45fda7e17329af76.mp4
2025-01-06 20:31:05,241 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json media/table/eval/total_success_rate_ci_6_6e461524546d8e890233.table.json
2025-01-06 20:31:05,241 INFO    SenderThread:2109502 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json
2025-01-06 20:31:05,241 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 10
2025-01-06 20:31:05,241 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:05,241 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:05,241 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-06 20:31:05,241 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:05,242 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-06 20:31:05,242 INFO    SenderThread:2109502 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 20:31:05,418 INFO    wandb-upload_0:2109502 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/requirements.txt
2025-01-06 20:31:05,425 INFO    wandb-upload_2:2109502 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/wandb-summary.json
2025-01-06 20:31:05,463 INFO    wandb-upload_4:2109502 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/output.log
2025-01-06 20:31:05,470 INFO    wandb-upload_1:2109502 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/files/config.yaml
2025-01-06 20:31:05,540 INFO    wandb-upload_3:2109502 [upload_job.py:push():130] Uploaded file /tmp/tmppey814qqwandb/5bwv6pdc-events.out.tfevents.1736213169.leonmkim-ROG-Strix-G15CS-G15CS.2109312.0
2025-01-06 20:31:05,741 INFO    Thread-11 (_thread_body):2109502 [sender.py:transition_state():614] send defer: 11
2025-01-06 20:31:05,741 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:05,741 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-06 20:31:05,741 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:05,741 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-06 20:31:05,741 INFO    SenderThread:2109502 [file_pusher.py:join():175] waiting for file pusher
2025-01-06 20:31:05,743 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 12
2025-01-06 20:31:05,743 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:05,743 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-06 20:31:05,743 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:05,743 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-06 20:31:05,743 INFO    SenderThread:2109502 [file_stream.py:finish():601] file stream finish called
2025-01-06 20:31:05,940 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:05,985 INFO    SenderThread:2109502 [file_stream.py:finish():605] file stream finish is done
2025-01-06 20:31:05,985 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 13
2025-01-06 20:31:05,985 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:05,985 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:05,985 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-06 20:31:05,985 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:05,985 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-06 20:31:05,985 INFO    SenderThread:2109502 [sender.py:transition_state():614] send defer: 14
2025-01-06 20:31:05,985 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: defer
2025-01-06 20:31:05,985 INFO    HandlerThread:2109502 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-06 20:31:05,986 DEBUG   SenderThread:2109502 [sender.py:send():379] send: final
2025-01-06 20:31:05,986 DEBUG   SenderThread:2109502 [sender.py:send():379] send: footer
2025-01-06 20:31:05,986 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: defer
2025-01-06 20:31:05,986 INFO    SenderThread:2109502 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-06 20:31:05,986 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:05,986 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:05,987 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: server_info
2025-01-06 20:31:05,987 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-06 20:31:05,987 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: get_summary
2025-01-06 20:31:05,987 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-06 20:31:05,987 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: server_info
2025-01-06 20:31:05,988 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-06 20:31:06,049 DEBUG   SenderThread:2109502 [sender.py:send_request():406] send_request: poll_exit
2025-01-06 20:31:06,050 INFO    MainThread:2109502 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-06 20:31:06,050 INFO    MainThread:2109502 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-06 20:31:06,050 INFO    MainThread:2109502 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-06 20:31:06,051 DEBUG   HandlerThread:2109502 [handler.py:handle_request():158] handle_request: shutdown
2025-01-06 20:31:06,051 INFO    HandlerThread:2109502 [handler.py:finish():882] shutting down handler
2025-01-06 20:31:06,987 INFO    WriterThread:2109502 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/yki2vgk2/wandb/run-20250106_202608-yki2vgk2/run-yki2vgk2.wandb
2025-01-06 20:31:07,050 INFO    SenderThread:2109502 [sender.py:finish():1615] shutting down sender
2025-01-06 20:31:07,050 INFO    SenderThread:2109502 [file_pusher.py:finish():169] shutting down file pusher
2025-01-06 20:31:07,050 INFO    SenderThread:2109502 [file_pusher.py:join():175] waiting for file pusher