2025-01-07 23:09:15,294 INFO    StreamThr :2249564 [internal.py:wandb_internal():85] W&B internal server running at pid: 2249564, started at: 2025-01-07 23:09:15.294013
2025-01-07 23:09:15,295 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status
2025-01-07 23:09:15,307 INFO    WriterThread:2249564 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/run-1evqtpqc.wandb
2025-01-07 23:09:15,308 DEBUG   SenderThread:2249564 [sender.py:send():379] send: header
2025-01-07 23:09:15,316 DEBUG   SenderThread:2249564 [sender.py:send():379] send: run
2025-01-07 23:09:15,559 INFO    SenderThread:2249564 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files
2025-01-07 23:09:15,559 INFO    SenderThread:2249564 [sender.py:_start_run_threads():1188] run started: 1evqtpqc with start time 1736309355.294227
2025-01-07 23:09:15,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: check_version
2025-01-07 23:09:15,578 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: check_version
2025-01-07 23:09:15,623 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: run_start
2025-01-07 23:09:15,628 DEBUG   HandlerThread:2249564 [system_info.py:__init__():26] System info init
2025-01-07 23:09:15,628 DEBUG   HandlerThread:2249564 [system_info.py:__init__():41] System info init done
2025-01-07 23:09:15,628 INFO    HandlerThread:2249564 [system_monitor.py:start():194] Starting system monitor
2025-01-07 23:09:15,628 INFO    SystemMonitor:2249564 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-07 23:09:15,629 INFO    HandlerThread:2249564 [system_monitor.py:probe():214] Collecting system info
2025-01-07 23:09:15,629 INFO    SystemMonitor:2249564 [interfaces.py:start():188] Started cpu monitoring
2025-01-07 23:09:15,629 INFO    SystemMonitor:2249564 [interfaces.py:start():188] Started disk monitoring
2025-01-07 23:09:15,630 INFO    SystemMonitor:2249564 [interfaces.py:start():188] Started gpu monitoring
2025-01-07 23:09:15,631 INFO    SystemMonitor:2249564 [interfaces.py:start():188] Started memory monitoring
2025-01-07 23:09:15,631 INFO    SystemMonitor:2249564 [interfaces.py:start():188] Started network monitoring
2025-01-07 23:09:15,641 DEBUG   HandlerThread:2249564 [system_info.py:probe():152] Probing system
2025-01-07 23:09:15,642 DEBUG   HandlerThread:2249564 [system_info.py:_probe_git():137] Probing git
2025-01-07 23:09:15,647 DEBUG   HandlerThread:2249564 [system_info.py:_probe_git():145] Probing git done
2025-01-07 23:09:15,647 DEBUG   HandlerThread:2249564 [system_info.py:probe():200] Probing system done
2025-01-07 23:09:15,647 DEBUG   HandlerThread:2249564 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T04:09:15.641385', 'startedAt': '2025-01-08T04:09:15.286608', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.159125, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.081, 'min': 800.0, 'max': 4700.0}, {'current': 4499.971, 'min': 800.0, 'max': 4700.0}, {'current': 4499.991, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 736.9039688110352}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-07 23:09:15,647 INFO    HandlerThread:2249564 [system_monitor.py:probe():224] Finished collecting system info
2025-01-07 23:09:15,647 INFO    HandlerThread:2249564 [system_monitor.py:probe():227] Publishing system info
2025-01-07 23:09:15,647 DEBUG   HandlerThread:2249564 [system_info.py:_save_code():44] Saving code
2025-01-07 23:09:15,649 DEBUG   HandlerThread:2249564 [system_info.py:_save_code():67] Saving code done
2025-01-07 23:09:15,649 DEBUG   HandlerThread:2249564 [system_info.py:_save_patches():84] Saving git patches
2025-01-07 23:09:15,710 DEBUG   HandlerThread:2249564 [system_info.py:_save_patches():126] Saving git patches done
2025-01-07 23:09:15,710 INFO    HandlerThread:2249564 [system_monitor.py:probe():229] Finished publishing system info
2025-01-07 23:09:15,713 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:09:15,714 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-07 23:09:15,714 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-07 23:09:15,714 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-07 23:09:15,941 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: python_packages
2025-01-07 23:09:15,941 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: python_packages
2025-01-07 23:09:15,942 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:09:15,942 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:09:15,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:15,977 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/ldm9zc23-wandb-metadata.json
2025-01-07 23:09:15,985 INFO    HandlerThread:2249564 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/tb"
  save: true
}
control {
  relay_id: "0e59b649b6e646d485666f1a9a652098"
}
_info {
  stream_id: "1evqtpqc"
}

2025-01-07 23:09:15,985 DEBUG   HandlerThread:2249564 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-07 23:09:16,013 INFO    wandb-upload_2:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/rtz14662-diff.patch
2025-01-07 23:09:16,031 DEBUG   SenderThread:2249564 [sender.py:send():379] send: telemetry
2025-01-07 23:09:16,035 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/erpa07wq-code/FISH/eval_robot.py
2025-01-07 23:09:16,036 DEBUG   SenderThread:2249564 [sender.py:send():379] send: tbrecord
2025-01-07 23:09:16,037 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:09:16,037 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0 with policy live
2025-01-07 23:09:16,242 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/chj417hk-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/diff.patch
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/requirements.txt
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/code/FISH/eval_robot.py
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-metadata.json
2025-01-07 23:09:16,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/code
2025-01-07 23:09:16,563 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/code/FISH
2025-01-07 23:09:16,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:17,943 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:18,560 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:18,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:19,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:20,943 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:20,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:21,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:22,788 DEBUG   SenderThread:2249564 [sender.py:send():379] send: config
2025-01-07 23:09:22,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:23,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:24,562 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:24,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:25,943 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:25,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:26,563 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:26,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:27,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:28,563 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:28,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:29,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:30,564 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:30,942 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:09:30,942 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:09:30,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:31,015 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:31,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:32,564 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:32,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:33,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:34,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:35,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:36,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:36,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:37,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:38,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:39,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:40,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:41,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:41,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:42,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:43,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:44,567 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:44,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:45,942 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:09:45,942 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:09:45,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:46,568 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:46,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:46,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:47,568 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/config.yaml
2025-01-07 23:09:47,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:48,569 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:09:48,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:49,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:50,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:51,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:52,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:52,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:53,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:54,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:55,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:56,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:57,944 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:09:57,949 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:58,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:09:59,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:00,942 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:10:00,942 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:10:00,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:01,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:02,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:03,021 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:03,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:04,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:05,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:06,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:07,951 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:08,879 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:08,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:09,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:10,574 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:10,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:11,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:12,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:13,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:14,418 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:14,575 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:14,624 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:10:14,624 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4 with policy now
2025-01-07 23:10:14,625 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:10:14,626 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:10:14,626 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:10:14,626 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:10:14,912 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/7hyyubkf-media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4
2025-01-07 23:10:14,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:15,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:10:15,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4
2025-01-07 23:10:15,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos
2025-01-07 23:10:15,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media
2025-01-07 23:10:15,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:10:15,631 DEBUG   SystemMonitor:2249564 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-07 23:10:15,632 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:10:15,874 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/y7kph9t8-media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4
2025-01-07 23:10:15,942 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:10:15,942 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:10:15,989 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:16,576 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:16,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:17,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:18,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:19,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:20,277 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:20,578 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:20,989 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:21,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:22,578 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:22,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:23,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:24,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:25,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:25,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:26,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:27,580 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:10:27,853 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/uabotbvt-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:10:27,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:28,580 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:28,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:29,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:30,581 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:30,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:10:30,947 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:10:30,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:30,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:31,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:32,582 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:32,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:33,954 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:34,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:35,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:36,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:36,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:37,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:38,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:39,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:40,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:41,946 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:41,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:42,955 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:43,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:44,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:45,633 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:10:45,945 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:10:45,945 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:10:45,989 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:46,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:47,080 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:47,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:48,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:49,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:50,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:51,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:52,345 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:52,588 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:52,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:53,956 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:54,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:55,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:56,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:57,472 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:10:57,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:58,590 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:10:58,730 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:10:58,730 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4 with policy now
2025-01-07 23:10:58,733 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:10:58,733 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:10:58,733 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:10:58,734 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:10:58,958 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:10:58,987 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/c1usdr64-media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4
2025-01-07 23:10:59,590 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:10:59,590 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4
2025-01-07 23:10:59,591 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:10:59,892 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/zladk6ho-media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4
2025-01-07 23:10:59,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:00,591 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:00,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:11:00,950 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:11:00,993 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:01,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:02,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:03,421 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:03,958 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:04,592 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:04,968 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:05,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:06,593 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:06,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:07,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:08,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:08,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:09,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:10,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:11,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:12,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:13,399 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:11:13,399 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:11:13,400 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:11:13,595 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:11:13,595 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:11:13,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:14,595 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:14,697 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:14,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:15,634 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:11:15,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:11:15,948 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:11:15,989 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:16,596 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:16,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:17,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:18,597 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:18,963 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:19,963 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:20,631 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:20,963 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:21,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:22,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:23,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:24,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:25,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:25,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:26,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:27,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:28,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:29,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:30,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:30,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:11:30,948 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:11:30,993 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:31,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:32,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:33,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:34,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:35,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:35,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:36,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:37,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:38,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:39,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:40,951 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:40,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:41,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:42,967 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:43,973 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:44,605 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:44,920 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:11:44,920 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4 with policy now
2025-01-07 23:11:44,921 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:11:44,921 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:11:44,921 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:11:44,922 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:11:44,971 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:45,173 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/a0bqb8rd-media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4
2025-01-07 23:11:45,606 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:11:45,606 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4
2025-01-07 23:11:45,606 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:11:45,634 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:11:45,899 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/gvum62ui-media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4
2025-01-07 23:11:45,950 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:11:45,951 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:11:45,993 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:46,037 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:46,606 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:46,973 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:47,606 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:47,975 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:48,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:49,980 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:50,607 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:50,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:51,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:51,978 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:52,608 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:11:52,982 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:53,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:54,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:55,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:56,958 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:11:56,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:57,976 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:58,979 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:11:59,982 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:00,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:12:00,953 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:12:00,997 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:01,238 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:12:01,238 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:12:01,239 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:12:01,611 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:12:01,611 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:12:01,897 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/382o6bh8-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:12:01,983 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:02,222 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:02,612 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:02,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:03,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:04,612 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:04,988 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:05,987 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:06,613 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:06,987 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:07,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:07,987 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:08,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:09,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:10,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:11,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:12,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:13,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:13,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:14,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:15,634 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:12:15,951 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:12:15,951 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:12:15,997 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:16,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:17,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:18,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:19,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:19,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:20,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:21,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:22,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:23,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:24,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:24,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:25,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:26,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:27,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:28,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:29,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:30,951 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:12:30,952 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:12:30,993 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:31,115 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:31,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:32,998 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:33,665 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:12:33,665 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4 with policy now
2025-01-07 23:12:33,668 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:12:33,668 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:12:33,668 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:12:33,670 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:12:33,957 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/imyk92rb-media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4
2025-01-07 23:12:33,997 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:34,622 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:12:34,622 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:34,623 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4
2025-01-07 23:12:34,623 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:12:34,921 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/49axge9l-media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4
2025-01-07 23:12:35,001 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:35,623 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:35,997 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:36,958 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:37,003 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:38,003 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:38,624 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:38,998 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:39,997 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:40,999 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:41,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:42,004 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:42,625 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:43,001 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:44,002 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:45,005 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:45,635 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:12:45,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:12:45,964 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:12:46,005 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:47,003 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:47,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:48,004 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:49,011 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:49,627 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:12:50,004 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:50,295 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:12:50,296 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:12:50,296 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:12:50,628 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:12:50,628 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:51,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:52,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:52,628 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:52,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:53,013 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:54,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:54,629 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:12:55,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:56,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:57,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:58,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:12:58,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:12:59,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:00,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:00,952 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:13:00,952 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:13:01,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:02,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:03,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:04,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:04,102 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:05,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:06,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:07,008 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:08,008 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:09,008 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:09,961 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:10,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:11,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:12,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:13,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:14,041 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:15,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:15,041 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:15,635 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:13:15,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:13:15,958 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:13:16,010 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:16,637 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:17,010 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:18,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:19,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:20,010 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:20,080 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:21,013 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:21,701 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:13:21,701 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4 with policy now
2025-01-07 23:13:21,705 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:13:21,705 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:13:21,705 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:13:21,706 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:13:21,953 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/mzs6vtga-media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4
2025-01-07 23:13:22,019 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:22,639 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:13:22,639 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:22,640 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4
2025-01-07 23:13:22,640 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:13:22,944 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/s6gkztjj-media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4
2025-01-07 23:13:23,020 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:24,018 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:25,023 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:25,347 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:26,023 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:26,641 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:27,024 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:28,033 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:29,024 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:30,024 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:30,642 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:30,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:13:30,962 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:13:31,026 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:31,095 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:32,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:33,029 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:34,046 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:35,043 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:36,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:36,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:37,038 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:38,047 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:38,645 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:13:38,895 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/7dbonkd8-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:13:39,047 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:39,355 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:13:39,356 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:13:39,356 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:13:39,645 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:13:40,056 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:40,645 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:41,052 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:41,976 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:42,050 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:43,049 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:44,049 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:44,647 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:45,057 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:45,636 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:13:45,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:13:45,957 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:13:46,057 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:47,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:47,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:48,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:49,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:50,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:51,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:52,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:52,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:53,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:54,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:54,651 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:13:55,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:56,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:57,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:57,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:13:58,067 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:13:59,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:00,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:00,957 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:14:00,957 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:14:01,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:02,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:03,031 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:03,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:04,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:05,069 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:06,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:07,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:08,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:08,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:09,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:10,075 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:10,569 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:14:10,570 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4 with policy now
2025-01-07 23:14:10,581 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:14:10,581 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:14:10,582 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:14:10,582 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:14:10,656 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:14:10,656 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:10,656 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4
2025-01-07 23:14:10,657 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:14:10,847 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/t0l6xdru-media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4
2025-01-07 23:14:11,094 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:11,123 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/iffolz4h-media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4
2025-01-07 23:14:12,086 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:12,657 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:13,087 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:13,973 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:14,092 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:15,091 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:15,636 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:14:15,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:14:15,962 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:14:16,091 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:16,658 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:17,094 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:18,101 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:18,659 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:19,117 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:19,169 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:20,109 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:21,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:22,115 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:22,410 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:14:22,410 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:14:22,410 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:14:22,660 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:14:22,660 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:14:23,104 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:24,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:24,269 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:24,661 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:25,117 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:26,106 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:27,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:28,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:28,663 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:29,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:29,282 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:30,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:30,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:14:30,962 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:14:31,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:32,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:33,108 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:34,109 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:34,629 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:35,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:36,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:36,665 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:37,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:38,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:38,666 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:39,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:40,074 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:40,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:41,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:42,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:43,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:44,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:45,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:45,637 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:14:45,638 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:45,958 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:14:45,958 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:14:46,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:47,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:48,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:49,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:50,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:50,968 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:51,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:52,111 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:53,116 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:54,131 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:54,531 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:14:54,531 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4 with policy now
2025-01-07 23:14:54,535 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:14:54,535 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:14:54,535 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:14:54,536 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:14:54,672 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:14:54,672 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:54,672 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4
2025-01-07 23:14:54,672 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:14:54,785 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/mvotjo30-media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4
2025-01-07 23:14:55,045 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/bnkkep60-media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4
2025-01-07 23:14:55,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:55,978 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:14:56,138 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:56,673 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:57,133 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:57,673 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:14:58,131 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:14:59,138 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:00,138 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:00,674 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:00,962 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:15:00,962 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:15:01,045 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:01,133 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:02,137 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:02,675 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:03,135 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:04,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:05,148 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:06,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:06,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:07,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:07,463 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:15:07,464 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:15:07,464 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:15:07,677 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:15:07,677 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:15:07,930 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/ki06q0ks-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:15:08,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:08,677 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:09,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:10,174 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:10,678 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:11,169 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:12,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:12,678 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:12,895 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:13,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:14,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:15,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:15,637 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:15:15,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:15:15,960 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:15:16,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:17,154 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:18,091 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:18,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:19,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:20,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:21,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:22,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:23,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:23,984 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:24,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:25,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:26,155 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:27,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:28,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:29,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:29,984 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:30,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:30,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:15:30,961 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:15:31,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:32,156 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:33,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:34,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:35,042 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:35,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:36,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:37,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:38,157 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:39,168 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:39,989 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:15:39,989 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4 with policy now
2025-01-07 23:15:39,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:15:39,992 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:15:39,992 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:15:39,992 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:15:40,042 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:40,172 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:40,247 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/vowh0va8-media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4
2025-01-07 23:15:40,688 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:15:40,689 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:40,689 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4
2025-01-07 23:15:40,689 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:15:40,974 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/m6a9zk1j-media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4
2025-01-07 23:15:41,190 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:41,689 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:42,185 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:42,690 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:43,190 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:44,184 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:44,690 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:45,187 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:45,513 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:45,638 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:15:45,980 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:15:45,980 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:15:46,189 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:47,182 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:48,186 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:48,692 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:15:49,186 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:50,202 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:50,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:51,198 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:52,190 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:53,194 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:54,210 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:55,214 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:55,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:15:56,218 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:57,216 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:58,219 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:15:59,222 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:00,217 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:00,977 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:16:00,977 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:16:01,064 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:01,218 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:02,229 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:03,224 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:04,221 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:05,223 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:06,223 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:07,001 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:07,226 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:08,230 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:09,229 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:10,232 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:11,236 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:12,002 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:12,233 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:13,238 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:14,251 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:15,244 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:15,638 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:16:15,970 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:16:15,970 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:16:16,249 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:17,104 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:17,243 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:18,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:19,249 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:20,270 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:21,266 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:22,255 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:23,007 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:23,264 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:24,258 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:25,265 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:26,268 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:27,263 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:28,019 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:28,259 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:29,266 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:30,279 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:30,966 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:16:30,966 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:16:31,276 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:32,281 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:33,077 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:33,274 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:34,275 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:35,280 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:36,283 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:37,283 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:38,284 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:39,022 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:39,280 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:40,281 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:41,297 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:42,302 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:43,300 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:44,025 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:44,311 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:45,303 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:45,639 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:16:45,971 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:16:45,972 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:16:46,297 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:47,298 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:48,317 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:49,083 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:49,326 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:50,349 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:50,592 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:16:50,592 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:16:50,593 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:16:50,714 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:16:50,714 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:16:51,331 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:52,310 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:52,715 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:16:53,336 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:54,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:54,327 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:16:55,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:56,324 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:56,716 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:16:57,340 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:58,328 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:59,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:16:59,658 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:00,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:00,718 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:00,963 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:17:00,963 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:17:01,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:02,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:03,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:04,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:04,719 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:04,991 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:05,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:06,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:07,329 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:08,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:08,721 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:09,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:10,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:10,641 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:11,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:12,330 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:13,331 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:14,331 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:15,331 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:15,639 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:17:15,963 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:17:15,964 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:17:16,046 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:16,337 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:17,334 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:18,335 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:19,335 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:20,335 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:21,335 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:22,026 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:22,336 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:23,336 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:24,336 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:25,336 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:26,345 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:26,593 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:17:26,593 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4 with policy now
2025-01-07 23:17:26,600 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:17:26,601 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:17:26,602 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:17:26,602 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:17:26,731 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:17:26,732 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:26,732 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4
2025-01-07 23:17:26,732 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:17:26,856 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/g2209p0a-media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4
2025-01-07 23:17:27,131 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/thr302eg-media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4
2025-01-07 23:17:27,360 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:27,616 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:28,368 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:28,732 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:29,370 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:30,377 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:30,733 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:30,971 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:17:30,972 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:17:31,363 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:32,371 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:33,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:33,363 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:34,380 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:34,736 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:35,368 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:36,390 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:37,372 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:38,042 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:38,381 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:39,390 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:40,373 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:41,391 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:42,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:42,652 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:17:42,652 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:17:42,653 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:17:42,739 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:17:42,739 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:42,739 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:17:43,002 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/d1jpx36j-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:17:43,387 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:43,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:44,388 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:44,740 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:45,388 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:45,640 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:17:45,969 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:17:45,969 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:17:46,395 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:46,741 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:47,381 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:48,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:48,742 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:17:49,073 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:49,381 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:50,381 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:51,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:52,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:53,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:54,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:55,045 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:17:55,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:56,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:57,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:58,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:17:59,382 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:00,047 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:00,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:00,965 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:18:00,966 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:18:01,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:02,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:03,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:04,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:05,094 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:05,383 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:06,384 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:07,384 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:08,384 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:09,384 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:10,385 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:11,045 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:11,385 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:12,385 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:13,385 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:14,351 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:18:14,351 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4 with policy now
2025-01-07 23:18:14,360 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:18:14,360 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:18:14,360 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:18:14,361 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:18:14,400 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:14,608 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/twgpqrl9-media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4
2025-01-07 23:18:14,752 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:18:14,752 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:14,752 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4
2025-01-07 23:18:14,752 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:18:15,028 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/rzsrf72v-media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4
2025-01-07 23:18:15,404 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:15,640 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:18:15,972 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:18:15,972 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:18:16,110 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:16,399 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:16,753 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:17,399 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:18,426 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:18,753 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:19,423 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:20,430 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:21,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:22,049 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:22,424 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:22,755 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:23,427 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:24,425 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:25,428 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:26,440 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:27,052 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:27,428 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:28,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:28,706 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:18:28,707 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:18:28,707 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:18:28,757 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:18:28,757 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:28,758 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:18:29,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:30,478 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:30,758 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:30,978 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:18:30,978 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:18:31,444 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:32,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:32,430 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:32,759 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:33,430 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:34,430 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:34,760 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:18:35,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:36,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:37,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:37,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:38,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:39,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:40,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:41,431 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:42,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:43,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:43,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:44,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:45,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:45,640 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:18:45,972 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:18:45,972 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:18:46,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:47,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:48,133 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:48,432 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:49,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:50,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:51,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:52,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:53,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:54,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:54,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:55,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:56,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:57,433 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:58,434 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:18:59,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:18:59,444 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:00,436 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:00,505 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:19:00,505 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4 with policy now
2025-01-07 23:19:00,505 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:19:00,506 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:19:00,506 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:19:00,506 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:19:00,767 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/s89wjrvk-media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4
2025-01-07 23:19:00,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:19:00,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:00,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4
2025-01-07 23:19:00,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:19:00,980 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:19:00,980 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:19:01,038 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/yb54cmhj-media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4
2025-01-07 23:19:01,461 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:01,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:02,461 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:02,770 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:03,451 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:04,129 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:04,452 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:04,771 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:05,457 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:06,468 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:06,772 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:07,461 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:08,469 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:08,773 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:09,185 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:09,486 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:10,495 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:11,488 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:12,473 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:13,492 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:13,760 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:19:13,761 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:19:13,762 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:19:13,775 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:19:13,775 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:19:14,025 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/r81g15bt-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:19:14,500 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:14,775 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:15,049 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:15,492 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:15,641 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:19:15,983 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:19:15,983 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:19:16,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:17,500 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:18,507 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:18,777 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:19,513 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:20,056 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:20,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:21,537 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:22,517 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:22,779 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:23,517 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:24,517 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:24,779 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:25,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:25,583 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:26,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:27,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:28,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:29,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:30,518 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:30,973 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:19:30,974 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:19:31,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:31,519 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:32,519 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:33,519 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:34,519 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:35,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:36,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:37,069 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:37,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:38,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:39,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:40,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:41,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:42,069 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:42,520 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:43,521 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:44,523 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:45,521 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:45,641 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:19:45,973 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:19:45,974 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:19:46,521 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:47,521 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:48,069 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:48,522 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:49,524 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:50,532 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:50,621 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:19:50,621 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4 with policy now
2025-01-07 23:19:50,661 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:19:50,661 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:19:50,662 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:19:50,662 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:19:50,789 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:19:50,789 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:50,790 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4
2025-01-07 23:19:50,790 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:19:50,872 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/bphbgl47-media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4
2025-01-07 23:19:51,143 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/eeo2xvy0-media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4
2025-01-07 23:19:51,530 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:52,541 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:52,790 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:53,084 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:53,552 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:53,790 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:54,551 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:55,556 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:56,543 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:56,792 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:57,542 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:58,457 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:19:58,557 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:19:58,792 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:19:59,571 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:00,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:01,000 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:20:01,000 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:20:01,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:02,562 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:02,794 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:02,794 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:20:02,818 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:20:02,818 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:20:02,818 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:20:03,497 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:03,555 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:03,794 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:20:04,573 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:04,795 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:05,564 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:06,574 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:07,569 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:08,534 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:08,562 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:08,796 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:09,570 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:10,582 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:11,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:12,582 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:13,586 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:13,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:14,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:14,799 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:15,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:15,642 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:20:15,976 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:20:15,977 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:20:16,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:17,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:18,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:19,105 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:19,577 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:20,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:21,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:22,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:23,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:24,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:25,082 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:25,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:26,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:27,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:28,578 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:29,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:30,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:30,976 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:20:30,977 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:20:31,106 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:31,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:32,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:33,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:34,579 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:35,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:36,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:37,083 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:37,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:38,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:39,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:40,580 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:41,611 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:41,869 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:20:41,870 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4 with policy now
2025-01-07 23:20:41,878 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:20:41,879 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:20:41,879 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:20:41,879 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:20:42,130 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/17k0r1ko-media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4
2025-01-07 23:20:42,606 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:42,810 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:20:42,810 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:42,810 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4
2025-01-07 23:20:42,811 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:20:42,964 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:43,146 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/h61tw7jt-media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4
2025-01-07 23:20:43,612 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:43,810 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:44,602 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:45,604 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:45,642 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:20:46,008 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:20:46,008 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:20:46,607 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:46,812 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:47,609 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:48,099 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:48,605 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:49,627 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:50,643 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:50,813 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:51,642 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:52,657 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:53,143 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:53,631 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:53,814 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:20:53,881 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:20:53,881 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:20:53,881 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:20:54,629 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:54,815 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:20:54,815 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:55,632 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:56,665 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:57,636 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:58,274 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:20:58,638 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:20:58,817 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:20:59,651 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:00,648 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:01,013 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:21:01,013 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:21:01,671 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:02,667 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:03,411 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:03,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:04,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:04,819 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:05,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:06,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:07,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:08,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:08,821 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:09,084 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:09,649 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:10,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:11,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:12,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:13,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:14,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:14,824 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:14,877 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:15,643 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:21:15,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:15,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:21:15,990 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:21:16,650 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:16,825 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:17,651 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:18,651 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:19,651 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:20,104 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:20,651 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:20,827 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:21,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:22,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:23,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:24,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:25,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:26,100 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:26,652 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:27,653 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:28,653 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:29,653 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:30,668 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:30,836 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:31,025 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:21:31,025 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:21:31,106 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:31,669 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:31,969 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:21:31,969 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4 with policy now
2025-01-07 23:21:31,985 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:21:31,985 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:21:31,986 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:21:31,986 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:21:32,226 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/3b41yzzt-media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4
2025-01-07 23:21:32,704 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:32,837 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:21:32,837 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:32,837 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4
2025-01-07 23:21:32,838 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:21:33,133 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/w7hak22a-media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4
2025-01-07 23:21:33,677 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:33,838 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:34,677 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:35,698 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:36,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:36,686 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:36,839 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:37,694 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:38,682 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:39,734 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:40,699 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:40,840 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:41,141 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:41,702 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:42,718 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:43,731 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:44,732 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:44,842 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:44,842 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:21:44,943 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:21:44,943 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:21:44,943 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:21:45,104 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/03z4z2vt-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:21:45,643 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:21:45,740 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:45,842 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:21:46,044 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:21:46,044 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:21:46,206 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:46,732 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:46,843 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:47,746 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:48,728 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:49,753 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:50,746 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:50,844 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:51,738 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:52,115 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:52,760 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:53,748 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:54,748 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:54,846 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:55,749 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:56,749 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:56,848 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:21:57,749 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:58,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:21:58,749 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:21:59,749 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:00,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:01,002 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:22:01,002 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:22:01,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:02,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:03,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:03,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:04,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:05,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:06,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:07,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:08,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:08,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:09,750 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:10,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:11,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:12,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:13,107 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:13,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:14,751 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:15,644 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:22:15,752 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:16,002 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:22:16,002 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:22:16,752 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:17,752 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:18,125 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:18,752 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:19,752 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:20,753 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:21,770 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:22,764 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:22,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:23,015 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:22:23,015 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4 with policy now
2025-01-07 23:22:23,033 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:22:23,034 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:22:23,034 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:22:23,034 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:22:23,283 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/do2pu68c-media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4
2025-01-07 23:22:23,757 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:23,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:22:23,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:23,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4
2025-01-07 23:22:23,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:22:24,120 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:24,159 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/ny7cwklp-media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4
2025-01-07 23:22:24,798 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:24,859 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:25,772 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:25,860 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:26,777 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:27,766 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:28,792 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:28,861 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:29,777 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:29,786 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:30,781 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:30,862 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:31,020 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:22:31,020 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:22:31,789 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:32,808 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:33,797 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:34,791 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:35,172 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:35,809 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:36,807 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:37,810 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:38,005 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:22:38,005 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:22:38,006 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:22:38,820 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:38,866 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:22:38,866 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:38,866 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:22:39,817 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:40,249 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:40,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:40,867 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:41,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:42,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:42,868 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:43,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:44,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:44,868 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:22:45,302 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:45,644 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:22:45,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:46,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:22:46,006 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:22:46,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:47,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:48,814 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:49,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:50,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:51,112 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:51,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:52,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:53,815 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:54,817 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:55,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:56,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:57,112 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:22:57,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:58,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:22:59,824 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:00,833 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:01,006 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:23:01,006 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:23:01,829 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:02,136 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:02,829 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:03,829 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:04,829 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:05,829 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:06,830 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:07,830 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:08,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:08,830 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:09,830 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:10,874 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:10,880 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:11,463 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:23:11,463 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4 with policy now
2025-01-07 23:23:11,490 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:23:11,490 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:23:11,491 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:23:11,491 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:23:11,712 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/dd30u4x4-media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4
2025-01-07 23:23:11,851 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:11,880 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:23:11,880 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:11,880 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4
2025-01-07 23:23:11,881 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:23:12,157 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/dgu28mn5-media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4
2025-01-07 23:23:12,881 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:12,887 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:13,595 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:13,881 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:13,885 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:14,865 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:15,644 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:23:15,888 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:16,026 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:23:16,026 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:23:16,882 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:16,900 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:17,896 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:18,874 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:18,883 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:19,501 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:19,891 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:20,884 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:20,891 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:21,896 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:22,878 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:22,885 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:22,885 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:23:23,063 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:23:23,063 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:23:23,064 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:23:23,885 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:23:23,905 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:24,513 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:24,886 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:24,896 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:25,900 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:26,898 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:27,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:28,887 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:28,906 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:29,653 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:29,913 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:30,908 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:31,056 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:23:31,057 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:23:31,922 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:32,932 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:33,941 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:34,934 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:35,164 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:35,916 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:36,891 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:36,916 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:37,926 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:38,922 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:39,922 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:40,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:41,131 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:41,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:42,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:43,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:44,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:45,645 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:23:45,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:46,013 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:23:46,013 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:23:46,166 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:46,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:47,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:48,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:49,923 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:50,924 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:51,924 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:52,131 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:52,924 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:53,924 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:54,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:55,926 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:56,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:57,132 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:23:57,899 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:23:57,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:58,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:23:59,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:00,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:01,014 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:24:01,015 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:24:01,926 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:02,133 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:02,947 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:03,964 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:24:03,964 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_01858989d758192b59d6.mp4 with policy now
2025-01-07 23:24:03,968 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:04,021 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:24:04,022 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:24:04,022 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:24:04,022 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:24:04,233 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/u2zwnre1-media/videos/eval/16_eval_16_01858989d758192b59d6.mp4
2025-01-07 23:24:04,911 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:24:04,912 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/16_eval_16_01858989d758192b59d6.mp4
2025-01-07 23:24:04,912 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:24:04,953 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:05,223 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/51cglfgr-media/videos/eval/16_eval_16_01858989d758192b59d6.mp4
2025-01-07 23:24:05,912 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:05,976 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:06,948 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:07,164 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:07,960 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:08,980 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:09,913 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:09,995 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:10,986 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:11,999 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:12,992 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:13,142 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:13,915 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:14,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:15,019 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:15,646 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:24:15,990 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:16,065 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:24:16,065 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:24:17,011 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:18,018 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:18,222 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:18,917 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:24:19,016 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:19,132 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:24:19,132 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:24:19,132 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:24:19,189 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/ye62zc23-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:24:19,918 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:24:19,918 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:20,023 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:21,014 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:21,919 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:22,005 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:23,014 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:23,785 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:24,014 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:25,038 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:25,920 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:26,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:27,051 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:28,063 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:29,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:29,164 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:30,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:31,057 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:24:31,057 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:24:31,105 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:32,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:33,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:33,924 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:34,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:34,835 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:35,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:36,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:37,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:38,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:39,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:40,009 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:40,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:41,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:41,928 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:24:42,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:43,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:44,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:45,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:45,165 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:45,646 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:24:46,031 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:24:46,031 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:24:46,077 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:47,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:48,035 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:49,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:50,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:50,186 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:51,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:52,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:53,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:54,036 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:55,037 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:56,038 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:56,162 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:24:57,038 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:58,038 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:59,053 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:24:59,942 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:00,038 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:25:00,038 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4 with policy now
2025-01-07 23:25:00,042 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:25:00,043 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:25:00,043 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:25:00,044 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:25:00,078 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:00,307 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/1s2u1h5y-media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4
2025-01-07 23:25:00,942 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:25:00,943 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4
2025-01-07 23:25:00,943 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:25:01,051 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:25:01,051 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:25:01,070 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:01,162 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:01,239 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/io1ylyl0-media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4
2025-01-07 23:25:01,943 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:02,067 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:03,069 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:04,084 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:05,080 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:05,944 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:06,096 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:06,174 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:07,068 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:08,073 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:09,085 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:09,946 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:10,095 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:11,113 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:11,176 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:11,194 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:25:11,194 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:25:11,194 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:25:11,947 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:25:11,947 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:25:12,128 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:13,108 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:13,948 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:14,109 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:15,121 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:15,647 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:25:16,044 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:25:16,044 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:25:16,098 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:16,180 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:17,106 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:17,950 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:18,095 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:19,119 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:20,102 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:21,119 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:22,144 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:22,175 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:23,122 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:24,132 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:25,125 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:26,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:27,122 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:27,954 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:28,122 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:28,172 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:29,122 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:29,955 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:30,122 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:31,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:25:31,034 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:25:31,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:32,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:33,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:33,669 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:33,957 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:34,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:35,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:36,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:37,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:38,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:39,123 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:39,176 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:40,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:41,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:42,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:43,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:44,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:45,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:45,174 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:45,647 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:25:46,034 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:25:46,035 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:25:46,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:47,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:48,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:49,124 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:50,125 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:50,174 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:51,125 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:52,126 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:53,139 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:53,966 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:54,140 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:54,341 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:25:54,341 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4 with policy now
2025-01-07 23:25:54,365 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:25:54,366 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:25:54,366 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:25:54,366 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:25:54,606 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/d4r27cpb-media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4
2025-01-07 23:25:54,967 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:25:54,967 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4
2025-01-07 23:25:54,967 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:25:55,148 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:55,287 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/m7ke0vqm-media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4
2025-01-07 23:25:55,452 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:25:55,968 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:25:56,151 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:57,175 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:58,173 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:59,163 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:25:59,969 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:00,185 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:00,925 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:01,078 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:26:01,078 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:26:01,177 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:02,187 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:03,186 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:03,971 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:04,205 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:05,189 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:06,207 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:06,218 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:07,205 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:07,973 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:26:08,212 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:08,266 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:26:08,266 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:26:08,266 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:26:08,973 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:26:08,973 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:26:09,229 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:09,974 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:10,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:11,221 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:11,241 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:11,974 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:12,227 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:13,243 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:13,975 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:14,272 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:15,248 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:15,648 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:26:16,075 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:26:16,075 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:26:16,231 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:16,266 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:17,249 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:18,271 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:19,265 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:20,249 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:21,239 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:21,285 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:22,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:23,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:23,980 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:24,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:25,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:26,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:27,191 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:27,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:28,250 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:29,251 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:30,258 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:31,050 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:26:31,050 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:26:31,251 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:32,191 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:32,251 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:33,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:34,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:35,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:36,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:37,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:37,733 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:37,986 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:38,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:39,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:40,254 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:41,252 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:42,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:42,930 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:43,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:43,989 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:44,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:45,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:45,648 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:26:46,050 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:26:46,050 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:26:46,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:47,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:48,191 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:48,253 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:49,310 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:49,991 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:50,276 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:50,296 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:26:50,296 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4 with policy now
2025-01-07 23:26:50,341 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:26:50,341 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:26:50,342 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:26:50,342 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:26:50,566 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/da34jd6f-media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4
2025-01-07 23:26:50,992 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:26:50,992 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4
2025-01-07 23:26:50,992 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval
2025-01-07 23:26:51,277 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:51,293 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/tp4pa9fo-media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4
2025-01-07 23:26:51,992 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:52,284 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:53,217 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:53,278 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:54,282 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:55,282 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:55,994 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:26:56,351 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:57,297 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:58,298 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:59,215 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:26:59,338 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:26:59,996 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:00,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:01,066 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:27:01,066 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:27:01,328 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:01,329 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:27:01,329 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:01,330 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:01,997 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:01,997 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:27:02,340 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:02,629 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 23:27:02,629 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 23:27:02,968 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:27:02,997 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json
2025-01-07 23:27:02,998 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media
2025-01-07 23:27:02,998 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval
2025-01-07 23:27:02,998 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table
2025-01-07 23:27:03,224 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/8ue5b6ml-media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json
2025-01-07 23:27:03,312 INFO    wandb-upload_0:2249564 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp6msy2ivh
2025-01-07 23:27:03,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:03,476 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-07 23:27:03,486 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: partial_history
2025-01-07 23:27:03,711 INFO    SenderThread:2249564 [sender.py:send_request_log_artifact():1525] logged artifact run-1evqtpqc-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjY0MjUxMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTkyNzQzOA==', 'latestArtifact': None}}
2025-01-07 23:27:03,711 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:27:03,711 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json with policy now
2025-01-07 23:27:03,712 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:27:03,712 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,713 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,713 DEBUG   SenderThread:2249564 [sender.py:send():379] send: telemetry
2025-01-07 23:27:03,713 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,713 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,713 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,715 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,715 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,716 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,716 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,716 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,717 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,717 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,717 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,717 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,718 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,718 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,718 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,719 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,719 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,719 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,719 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,720 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,720 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:03,720 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:03,720 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: log_artifact
2025-01-07 23:27:03,998 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:03,999 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:03,999 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:27:03,999 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json
2025-01-07 23:27:04,000 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval
2025-01-07 23:27:04,002 INFO    wandb-upload_2:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/x9oxhdwl-media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json
2025-01-07 23:27:04,234 INFO    wandb-upload_2:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/at5tb6y0-media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json
2025-01-07 23:27:04,282 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/vhynu0g9-events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:27:04,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:04,518 INFO    wandb-upload_0:2249564 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1xjl6479
2025-01-07 23:27:04,898 INFO    SenderThread:2249564 [sender.py:send_request_log_artifact():1525] logged artifact run-1evqtpqc-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjY0MjUyMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTkyNzQ0MQ==', 'latestArtifact': None}}
2025-01-07 23:27:04,901 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:05,013 DEBUG   SenderThread:2249564 [sender.py:send():379] send: files
2025-01-07 23:27:05,014 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json with policy now
2025-01-07 23:27:05,014 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:27:05,014 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:05,014 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:05,291 INFO    wandb-upload_2:2249564 [upload_job.py:push():130] Uploaded file /tmp/tmpwboe2p40wandb/rg32mlkp-media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json
2025-01-07 23:27:05,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:06,017 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:06,017 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:06,018 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/config.yaml
2025-01-07 23:27:06,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:07,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:08,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:09,319 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:10,220 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:10,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:11,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:12,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:13,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:14,320 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:15,220 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:15,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:15,649 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:27:16,055 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: stop_status
2025-01-07 23:27:16,055 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: stop_status
2025-01-07 23:27:16,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:17,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:18,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:19,321 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:20,053 DEBUG   SenderThread:2249564 [sender.py:send():379] send: exit
2025-01-07 23:27:20,053 INFO    SenderThread:2249564 [sender.py:send_exit():586] handling exit code: 0
2025-01-07 23:27:20,054 INFO    SenderThread:2249564 [sender.py:send_exit():588] handling runtime: 1084
2025-01-07 23:27:20,054 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:20,054 INFO    SenderThread:2249564 [sender.py:send_exit():594] send defer
2025-01-07 23:27:20,054 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:20,054 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-07 23:27:20,054 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:20,054 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-07 23:27:20,054 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 1
2025-01-07 23:27:20,055 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:20,055 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-07 23:27:20,055 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:20,055 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-07 23:27:20,055 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 2
2025-01-07 23:27:20,055 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:20,055 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-07 23:27:20,055 INFO    HandlerThread:2249564 [system_monitor.py:finish():203] Stopping system monitor
2025-01-07 23:27:20,055 DEBUG   SystemMonitor:2249564 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-07 23:27:20,055 DEBUG   SystemMonitor:2249564 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-07 23:27:20,055 INFO    HandlerThread:2249564 [interfaces.py:finish():200] Joined cpu monitor
2025-01-07 23:27:20,056 INFO    HandlerThread:2249564 [interfaces.py:finish():200] Joined disk monitor
2025-01-07 23:27:20,107 INFO    HandlerThread:2249564 [interfaces.py:finish():200] Joined gpu monitor
2025-01-07 23:27:20,108 INFO    HandlerThread:2249564 [interfaces.py:finish():200] Joined memory monitor
2025-01-07 23:27:20,108 INFO    HandlerThread:2249564 [interfaces.py:finish():200] Joined network monitor
2025-01-07 23:27:20,108 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:20,108 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-07 23:27:20,108 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 3
2025-01-07 23:27:20,108 DEBUG   SenderThread:2249564 [sender.py:send():379] send: stats
2025-01-07 23:27:20,108 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:20,109 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-07 23:27:20,109 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:20,109 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-07 23:27:20,109 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 4
2025-01-07 23:27:20,109 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:20,109 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-07 23:27:21,025 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:22,025 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:26,340 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 23:27:26,340 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:26,340 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: keepalive
2025-01-07 23:27:26,341 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:26,341 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:26,342 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-07 23:27:26,342 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 5
2025-01-07 23:27:26,342 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 23:27:26,342 DEBUG   SenderThread:2249564 [sender.py:send():379] send: history
2025-01-07 23:27:26,342 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:26,342 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-07 23:27:26,342 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: summary_record
2025-01-07 23:27:26,343 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:26,343 DEBUG   SenderThread:2249564 [sender.py:send():379] send: summary
2025-01-07 23:27:26,344 INFO    SenderThread:2249564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-07 23:27:26,344 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:26,344 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-07 23:27:26,344 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 6
2025-01-07 23:27:26,344 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:26,344 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-07 23:27:26,344 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:26,344 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-07 23:27:26,345 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 7
2025-01-07 23:27:26,345 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: status_report
2025-01-07 23:27:26,345 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:26,345 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-07 23:27:26,345 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:26,345 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-07 23:27:27,028 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:27,055 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 23:27:27,613 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 8
2025-01-07 23:27:27,613 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 23:27:27,614 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:27,614 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-07 23:27:27,614 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:27,614 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-07 23:27:27,614 INFO    SenderThread:2249564 [job_builder.py:build():440] Attempting to build job artifact
2025-01-07 23:27:27,616 INFO    SenderThread:2249564 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-07 23:27:27,618 INFO    SenderThread:2249564 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-07 23:27:27,629 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 9
2025-01-07 23:27:27,630 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:27,630 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-07 23:27:27,630 DEBUG   SenderThread:2249564 [sender.py:send():379] send: artifact
2025-01-07 23:27:27,885 INFO    SenderThread:2249564 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzUzMw==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzUzMw==', 'versionIndex': 439}}}
2025-01-07 23:27:27,885 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:27,885 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-07 23:27:27,885 INFO    SenderThread:2249564 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-07 23:27:28,028 INFO    Thread-12 :2249564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:28,029 INFO    SenderThread:2249564 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files
2025-01-07 23:27:28,029 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/config.yaml config.yaml
2025-01-07 23:27:28,029 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json wandb-summary.json
2025-01-07 23:27:28,030 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/requirements.txt requirements.txt
2025-01-07 23:27:28,030 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-metadata.json wandb-metadata.json
2025-01-07 23:27:28,030 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0 events.out.tfevents.1736309355.leonmkim-ROG-Strix-G15CS-G15CS.2249399.0
2025-01-07 23:27:28,030 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log output.log
2025-01-07 23:27:28,030 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/diff.patch diff.patch
2025-01-07 23:27:28,031 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-07 23:27:28,037 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/16_eval_16_01858989d758192b59d6.mp4 media/videos/eval/16_eval_16_01858989d758192b59d6.mp4
2025-01-07 23:27:28,037 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4 media/videos/eval/2_eval_2_b724e203bcdc67aa5a2f.mp4
2025-01-07 23:27:28,037 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4 media/videos/eval/3_eval_3_7a1f2288b71555e11681.mp4
2025-01-07 23:27:28,037 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4 media/videos/eval/1_eval_1_a6cba8accd20e1b8ec9e.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4 media/videos/eval/14_eval_14_e423f85e531e4aa2eda4.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4 media/videos/eval/0_eval_0_da5d8645307c977c2178.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4 media/videos/eval/4_eval_4_642bdc6d95c526c17f07.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4 media/videos/eval/12_eval_12_84f1b7d03de03c92ec2c.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4 media/videos/eval/9_eval_9_811955fe3a96368a3ead.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4 media/videos/eval/15_eval_15_6b8c887cc79fe4bb5aaf.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4 media/videos/eval/8_eval_8_1b915a87c852375007f7.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4 media/videos/eval/7_eval_7_9c6581231703b0f374d0.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4 media/videos/eval/13_eval_13_b9f4d6e4d981133c66e5.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4 media/videos/eval/11_eval_11_c3e8fea09465a60cb8c5.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4 media/videos/eval/5_eval_5_465c60fb8be6cb25f820.mp4
2025-01-07 23:27:28,038 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4 media/videos/eval/17_eval_17_1c3c2ba281fd3f6a3733.mp4
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4 media/videos/eval/18_eval_18_3d9fd71a9a5e8f25cb3e.mp4
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4 media/videos/eval/19_eval_19_49104c03ad255c9ff4c0.mp4
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4 media/videos/eval/10_eval_10_b875e5dd12f721236870.mp4
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4 media/videos/eval/6_eval_6_df7000ad25a0ae7e07a9.mp4
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json media/table/eval/success_rate_ci_20_6d6e13da9233b1fe7784.table.json
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json media/table/eval/total_success_rate_ci_21_f8b20affcaa8e1150c3b.table.json
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 10
2025-01-07 23:27:28,039 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:28,039 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-07 23:27:28,039 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-07 23:27:28,039 INFO    SenderThread:2249564 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 23:27:28,055 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 23:27:28,055 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 23:27:28,214 INFO    wandb-upload_3:2249564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/config.yaml
2025-01-07 23:27:28,227 INFO    wandb-upload_0:2249564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/wandb-summary.json
2025-01-07 23:27:28,243 INFO    wandb-upload_2:2249564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/output.log
2025-01-07 23:27:28,252 INFO    wandb-upload_1:2249564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/files/requirements.txt
2025-01-07 23:27:28,453 INFO    Thread-11 (_thread_body):2249564 [sender.py:transition_state():614] send defer: 11
2025-01-07 23:27:28,453 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:28,453 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-07 23:27:28,453 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:28,453 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-07 23:27:28,453 INFO    SenderThread:2249564 [file_pusher.py:join():175] waiting for file pusher
2025-01-07 23:27:28,456 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 12
2025-01-07 23:27:28,457 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:28,457 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-07 23:27:28,457 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:28,457 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-07 23:27:28,457 INFO    SenderThread:2249564 [file_stream.py:finish():601] file stream finish called
2025-01-07 23:27:28,655 INFO    SenderThread:2249564 [file_stream.py:finish():605] file stream finish is done
2025-01-07 23:27:28,655 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 13
2025-01-07 23:27:28,655 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:28,655 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-07 23:27:28,656 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:28,656 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-07 23:27:28,656 INFO    SenderThread:2249564 [sender.py:transition_state():614] send defer: 14
2025-01-07 23:27:28,656 DEBUG   SenderThread:2249564 [sender.py:send():379] send: final
2025-01-07 23:27:28,656 DEBUG   SenderThread:2249564 [sender.py:send():379] send: footer
2025-01-07 23:27:28,656 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: defer
2025-01-07 23:27:28,656 INFO    HandlerThread:2249564 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-07 23:27:28,656 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: defer
2025-01-07 23:27:28,656 INFO    SenderThread:2249564 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-07 23:27:28,657 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 23:27:28,657 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 23:27:28,657 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: server_info
2025-01-07 23:27:28,657 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-07 23:27:28,658 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: server_info
2025-01-07 23:27:28,658 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: get_summary
2025-01-07 23:27:28,660 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-07 23:27:28,661 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-07 23:27:28,708 DEBUG   SenderThread:2249564 [sender.py:send_request():406] send_request: poll_exit
2025-01-07 23:27:28,708 INFO    MainThread:2249564 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-07 23:27:28,709 INFO    MainThread:2249564 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-07 23:27:28,709 INFO    MainThread:2249564 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-07 23:27:28,709 DEBUG   HandlerThread:2249564 [handler.py:handle_request():158] handle_request: shutdown
2025-01-07 23:27:28,709 INFO    HandlerThread:2249564 [handler.py:finish():882] shutting down handler
2025-01-07 23:27:29,658 INFO    WriterThread:2249564 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/1evqtpqc/wandb/run-20250107_230915-1evqtpqc/run-1evqtpqc.wandb
2025-01-07 23:27:29,708 INFO    SenderThread:2249564 [sender.py:finish():1615] shutting down sender
2025-01-07 23:27:29,708 INFO    SenderThread:2249564 [file_pusher.py:finish():169] shutting down file pusher
2025-01-07 23:27:29,708 INFO    SenderThread:2249564 [file_pusher.py:join():175] waiting for file pusher