2025-01-18 21:03:44,037 INFO    StreamThr :2577265 [internal.py:wandb_internal():85] W&B internal server running at pid: 2577265, started at: 2025-01-18 21:03:44.037545
2025-01-18 21:03:44,039 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status
2025-01-18 21:03:44,053 INFO    WriterThread:2577265 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/run-2t6elatx.wandb
2025-01-18 21:03:44,054 DEBUG   SenderThread:2577265 [sender.py:send():379] send: header
2025-01-18 21:03:44,061 DEBUG   SenderThread:2577265 [sender.py:send():379] send: run
2025-01-18 21:03:44,485 INFO    SenderThread:2577265 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files
2025-01-18 21:03:44,485 INFO    SenderThread:2577265 [sender.py:_start_run_threads():1188] run started: 2t6elatx with start time 1737252224.03754
2025-01-18 21:03:44,500 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: check_version
2025-01-18 21:03:44,500 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: check_version
2025-01-18 21:03:44,548 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: run_start
2025-01-18 21:03:44,553 DEBUG   HandlerThread:2577265 [system_info.py:__init__():26] System info init
2025-01-18 21:03:44,553 DEBUG   HandlerThread:2577265 [system_info.py:__init__():41] System info init done
2025-01-18 21:03:44,553 INFO    HandlerThread:2577265 [system_monitor.py:start():194] Starting system monitor
2025-01-18 21:03:44,554 INFO    SystemMonitor:2577265 [system_monitor.py:_start():158] Starting system asset monitoring threads
2025-01-18 21:03:44,554 INFO    HandlerThread:2577265 [system_monitor.py:probe():214] Collecting system info
2025-01-18 21:03:44,554 INFO    SystemMonitor:2577265 [interfaces.py:start():188] Started cpu monitoring
2025-01-18 21:03:44,555 INFO    SystemMonitor:2577265 [interfaces.py:start():188] Started disk monitoring
2025-01-18 21:03:44,555 INFO    SystemMonitor:2577265 [interfaces.py:start():188] Started gpu monitoring
2025-01-18 21:03:44,556 INFO    SystemMonitor:2577265 [interfaces.py:start():188] Started memory monitoring
2025-01-18 21:03:44,557 INFO    SystemMonitor:2577265 [interfaces.py:start():188] Started network monitoring
2025-01-18 21:03:44,567 DEBUG   HandlerThread:2577265 [system_info.py:probe():152] Probing system
2025-01-18 21:03:44,568 DEBUG   HandlerThread:2577265 [system_info.py:_probe_git():137] Probing git
2025-01-18 21:03:44,573 DEBUG   HandlerThread:2577265 [system_info.py:_probe_git():145] Probing git done
2025-01-18 21:03:44,573 DEBUG   HandlerThread:2577265 [system_info.py:probe():200] Probing system done
2025-01-18 21:03:44,573 DEBUG   HandlerThread:2577265 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-19T02:03:44.567264', 'startedAt': '2025-01-19T02:03:44.029624', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3558.444375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4480.37, 'min': 800.0, 'max': 4700.0}, {'current': 4488.089, 'min': 800.0, 'max': 4700.0}, {'current': 4562.354, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 658.0720520019531}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}}
2025-01-18 21:03:44,573 INFO    HandlerThread:2577265 [system_monitor.py:probe():224] Finished collecting system info
2025-01-18 21:03:44,573 INFO    HandlerThread:2577265 [system_monitor.py:probe():227] Publishing system info
2025-01-18 21:03:44,573 DEBUG   HandlerThread:2577265 [system_info.py:_save_code():44] Saving code
2025-01-18 21:03:44,574 DEBUG   HandlerThread:2577265 [system_info.py:_save_code():67] Saving code done
2025-01-18 21:03:44,575 DEBUG   HandlerThread:2577265 [system_info.py:_save_patches():84] Saving git patches
2025-01-18 21:03:44,669 DEBUG   HandlerThread:2577265 [system_info.py:_save_patches():126] Saving git patches done
2025-01-18 21:03:44,670 INFO    HandlerThread:2577265 [system_monitor.py:probe():229] Finished publishing system info
2025-01-18 21:03:44,673 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:03:44,673 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now
2025-01-18 21:03:44,673 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now
2025-01-18 21:03:44,673 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file diff.patch with policy now
2025-01-18 21:03:44,674 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now
2025-01-18 21:03:44,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: python_packages
2025-01-18 21:03:44,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:03:44,904 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: python_packages
2025-01-18 21:03:44,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:03:44,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:44,906 INFO    HandlerThread:2577265 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord {
  log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/tb"
  save: true
}
control {
  relay_id: "bcf15667ba81445d81be1061d1a1388d"
}
_info {
  stream_id: "2t6elatx"
}

2025-01-18 21:03:44,907 DEBUG   HandlerThread:2577265 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml
2025-01-18 21:03:44,931 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/a4oe7r2l-wandb-metadata.json
2025-01-18 21:03:44,938 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/cl0olw9t-code/FISH/eval_robot.py
2025-01-18 21:03:44,946 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/h5brxswo-diff.patch
2025-01-18 21:03:44,949 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/f3we0mxa-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch
2025-01-18 21:03:45,019 DEBUG   SenderThread:2577265 [sender.py:send():379] send: telemetry
2025-01-18 21:03:45,019 DEBUG   SenderThread:2577265 [sender.py:send():379] send: tbrecord
2025-01-18 21:03:45,020 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:03:45,020 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0 with policy live
2025-01-18 21:03:45,201 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/vu00izql-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:03:45,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/requirements.txt
2025-01-18 21:03:45,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:03:45,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/code/FISH/eval_robot.py
2025-01-18 21:03:45,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch
2025-01-18 21:03:45,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/diff.patch
2025-01-18 21:03:45,488 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:03:45,488 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-metadata.json
2025-01-18 21:03:45,488 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/code/FISH
2025-01-18 21:03:45,488 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/code
2025-01-18 21:03:45,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:46,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:47,487 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:03:47,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:48,880 DEBUG   SenderThread:2577265 [sender.py:send():379] send: config
2025-01-18 21:03:48,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:49,889 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:03:49,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:50,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:51,488 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:03:51,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:52,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:53,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:54,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:03:54,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:55,489 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:03:55,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:56,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:57,489 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:03:57,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:58,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:03:59,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:03:59,904 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:03:59,945 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:00,046 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:00,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:01,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:02,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:03,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:04,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:05,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:05,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:06,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:07,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:08,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:09,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:10,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:11,493 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:11,637 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:11,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:12,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:13,494 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:13,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:14,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:04:14,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:04:14,945 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:15,494 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:15,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:16,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:16,986 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:17,495 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/config.yaml
2025-01-18 21:04:17,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:18,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:19,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:20,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:21,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:22,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:22,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:23,911 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:24,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:25,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:26,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:27,906 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:27,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:28,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:29,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:04:29,904 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:04:29,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:30,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:31,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:32,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:33,058 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:33,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:34,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:35,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:36,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:37,914 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:38,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:38,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:39,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:40,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:41,240 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:04:41,240 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4 with policy now
2025-01-18 21:04:41,240 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:04:41,241 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:04:41,241 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:04:41,242 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:04:41,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos
2025-01-18 21:04:41,508 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/a4g7ag7y-media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4
2025-01-18 21:04:41,799 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/9nlf524s-media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4
2025-01-18 21:04:41,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:42,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:43,503 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:43,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:44,252 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:44,557 DEBUG   SystemMonitor:2577265 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2025-01-18 21:04:44,558 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:04:44,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:04:44,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:04:44,916 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:45,917 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:46,916 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:47,505 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:47,916 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:48,917 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:49,505 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:49,609 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:49,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:50,917 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:51,917 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:52,937 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:53,507 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:04:53,768 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/i4lnxezk-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:04:53,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:54,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:04:54,917 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:55,507 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:55,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:56,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:57,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:58,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:04:59,509 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:04:59,904 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:04:59,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:04:59,945 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:00,029 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:00,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:01,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:02,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:03,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:04,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:05,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:05,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:06,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:07,919 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:08,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:09,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:10,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:10,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:11,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:12,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:13,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:14,566 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:05:14,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:05:14,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:05:14,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:15,514 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:15,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:15,990 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:16,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:17,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:18,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:19,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:20,922 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:21,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:21,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:22,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:23,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:24,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:25,145 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:05:25,146 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:05:25,146 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4 with policy now
2025-01-18 21:05:25,146 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:05:25,146 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:05:25,146 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:05:25,416 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/u5p126k7-media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4
2025-01-18 21:05:25,517 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:25,517 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:05:25,517 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4
2025-01-18 21:05:25,517 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:05:25,822 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/jpfn1j2z-media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4
2025-01-18 21:05:25,926 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:26,925 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:27,151 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:27,518 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:27,926 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:28,927 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:29,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:05:29,910 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:05:29,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:30,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:31,519 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:31,926 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:32,573 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:32,933 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:33,520 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:33,931 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:34,313 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:05:34,313 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:05:34,313 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:05:34,520 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:05:34,520 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:05:34,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:35,521 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:35,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:36,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:37,521 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:37,636 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:37,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:38,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:39,522 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:39,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:40,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:41,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:42,911 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:42,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:43,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:44,559 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:05:44,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:05:44,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:05:44,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:45,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:46,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:47,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:47,993 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:48,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:49,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:50,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:51,526 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:05:51,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:52,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:53,911 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:05:53,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:54,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:55,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:56,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:57,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:58,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:59,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:05:59,905 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:05:59,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:05:59,997 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:00,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:01,931 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:02,931 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:03,932 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:04,932 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:05,193 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:05,941 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:06,234 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:06:06,234 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_9019146f9223df893e27.mp4 with policy now
2025-01-18 21:06:06,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:06:06,237 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:06:06,237 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:06:06,238 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:06:06,531 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:06:06,531 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/2_eval_2_9019146f9223df893e27.mp4
2025-01-18 21:06:06,531 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:06:06,538 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/8kjphwxb-media/videos/eval/2_eval_2_9019146f9223df893e27.mp4
2025-01-18 21:06:06,866 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/c5b0q6f7-media/videos/eval/2_eval_2_9019146f9223df893e27.mp4
2025-01-18 21:06:06,934 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:07,531 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:07,937 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:08,937 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:09,938 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:10,921 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:10,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:11,532 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:11,938 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:12,939 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:13,533 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:13,943 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:14,559 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:06:14,934 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:06:14,934 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:06:14,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:15,534 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:15,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:16,004 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:16,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:17,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:18,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:19,535 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:06:19,828 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/5sqph8os-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:06:19,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:20,151 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:06:20,151 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:06:20,151 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:06:20,536 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:06:20,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:21,536 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:21,557 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:21,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:22,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:23,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:24,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:25,537 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:25,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:26,922 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:26,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:27,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:28,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:29,539 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:29,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:06:29,920 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:06:29,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:30,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:31,539 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:31,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:32,070 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:32,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:33,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:34,950 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:35,951 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:36,951 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:37,225 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:37,542 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:37,951 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:38,951 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:39,951 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:40,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:41,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:42,922 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:42,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:43,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:44,560 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:06:44,918 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:06:44,919 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:06:44,961 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:45,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:46,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:47,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:48,006 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:48,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:49,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:50,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:51,956 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:52,430 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:06:52,430 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4 with policy now
2025-01-18 21:06:52,433 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:06:52,433 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:06:52,433 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:06:52,434 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:06:52,557 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:06:52,558 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4
2025-01-18 21:06:52,558 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:06:52,694 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/6be1rgm0-media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4
2025-01-18 21:06:52,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:52,989 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/2gh50kbn-media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4
2025-01-18 21:06:53,454 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:53,558 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:53,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:54,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:55,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:56,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:57,559 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:06:57,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:58,871 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:06:58,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:06:59,925 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:06:59,925 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:06:59,965 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:00,968 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:01,560 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:01,966 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:02,975 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:03,972 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:04,065 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:04,974 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:05,973 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:06,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:07,207 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:07:07,207 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:07:07,207 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:07:07,562 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:07:07,563 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:07:07,968 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:08,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:09,117 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:09,563 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:09,982 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:10,974 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:11,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:12,972 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:13,565 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:13,973 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:14,284 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:14,560 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:07:14,920 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:07:14,920 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:07:14,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:15,565 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:15,973 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:16,973 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:17,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:18,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:19,932 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:19,975 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:20,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:21,975 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:22,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:23,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:24,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:25,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:25,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:26,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:27,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:28,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:29,923 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:07:29,923 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:07:29,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:30,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:31,006 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:31,977 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:32,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:33,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:34,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:35,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:36,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:36,981 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:37,979 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:38,979 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:39,980 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:40,992 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:41,097 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:07:41,097 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4 with policy now
2025-01-18 21:07:41,098 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:07:41,098 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:07:41,099 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:07:41,100 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:07:41,367 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/29jfdqu6-media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4
2025-01-18 21:07:41,574 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:41,575 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:07:41,575 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4
2025-01-18 21:07:41,575 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:07:41,873 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/yjzneliy-media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4
2025-01-18 21:07:41,989 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:42,133 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:42,989 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:43,575 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:43,986 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:44,561 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:07:44,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:07:44,949 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:07:44,989 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:45,576 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:45,992 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:46,991 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:47,611 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:47,986 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:48,990 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:49,577 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:50,006 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:51,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:52,004 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:52,260 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:07:52,260 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:07:52,262 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:07:52,578 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:07:52,579 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:07:52,843 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/r19j16bi-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:07:52,876 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:53,009 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:53,579 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:54,004 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:55,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:55,579 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:56,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:56,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:57,580 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:07:57,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:58,247 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:07:58,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:07:59,923 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:07:59,923 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:07:59,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:00,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:01,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:03,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:04,000 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:04,077 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:05,000 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:06,000 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:07,000 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:08,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:09,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:09,932 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:10,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:11,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:12,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:13,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:14,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:14,561 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:08:14,923 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:08:14,923 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:08:15,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:15,012 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:16,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:17,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:18,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:19,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:20,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:20,933 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:21,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:22,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:23,005 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:24,009 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:24,117 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:08:24,117 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4 with policy now
2025-01-18 21:08:24,123 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:08:24,123 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:08:24,123 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:08:24,124 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:08:24,382 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/4bra0h1v-media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4
2025-01-18 21:08:24,591 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:08:24,591 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4
2025-01-18 21:08:24,591 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:08:24,884 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/uxndxhhz-media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4
2025-01-18 21:08:25,021 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:25,591 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:25,941 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:26,022 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:27,023 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:28,024 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:29,020 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:29,592 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:29,928 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:08:29,928 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:08:30,028 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:31,013 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:31,593 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:31,709 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:32,026 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:33,028 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:33,594 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:08:34,026 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:34,312 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:08:34,313 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:08:34,314 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:08:34,594 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:08:34,594 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:08:35,029 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:35,595 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:36,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:36,941 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:37,041 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:38,049 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:39,042 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:39,596 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:40,040 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:41,041 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:42,029 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:42,513 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:43,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:43,597 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:08:44,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:44,561 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:08:44,927 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:08:44,927 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:08:45,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:46,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:47,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:48,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:48,036 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:49,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:50,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:51,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:52,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:53,031 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:53,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:08:54,032 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:55,032 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:56,032 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:57,032 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:58,032 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:59,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:08:59,927 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:08:59,927 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:08:59,994 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:00,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:01,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:02,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:03,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:04,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:05,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:05,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:06,034 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:07,034 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:08,034 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:09,034 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:10,042 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:10,477 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:09:10,478 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4 with policy now
2025-01-18 21:09:10,486 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:09:10,486 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:09:10,486 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:09:10,487 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:09:10,607 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:09:10,608 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4
2025-01-18 21:09:10,608 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:09:10,751 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/2bl04fqg-media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4
2025-01-18 21:09:11,050 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:11,078 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/mxlj1ylo-media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4
2025-01-18 21:09:11,534 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:11,608 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:12,047 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:13,053 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:14,061 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:14,562 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:09:14,955 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:09:14,955 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:09:15,056 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:15,609 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:16,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:17,026 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:17,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:18,092 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:19,082 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:19,611 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:20,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:21,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:22,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:22,378 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:09:22,378 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:09:22,378 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:22,378 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:09:22,612 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:09:22,612 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:09:22,873 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/c7lpznkb-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:09:23,095 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:23,612 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:24,092 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:25,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:25,613 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:26,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:27,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:27,614 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:28,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:28,321 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:29,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:29,615 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:29,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:09:29,930 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:09:30,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:31,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:32,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:33,085 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:34,015 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:34,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:35,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:36,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:37,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:38,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:39,086 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:39,948 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:40,088 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:41,089 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:42,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:43,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:44,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:44,562 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:09:44,570 WARNING FileStreamThread:2577265 [file_stream.py:request_with_retry():674] requests_with_retry encountered retryable exception: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). func: <bound method Session.post of <requests.sessions.Session object at 0x7faa5b95f640>>, args: ('https://api.wandb.ai/files/serialexperimentsleon/extrinsic_contact_downstream/2t6elatx/file_stream',), kwargs: {'json': {'files': {'wandb-summary.json': {'offset': 0, 'content': ['{"eval/0_eval": {"_type": "video-file", "sha256": "373935c9a53d50a21516ae9864a2b682a53224d448bd3a5444b7c73d070560df", "size": 1151995, "path": "media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4"}, "global_step": 796, "_timestamp": 1737252513.5501094, "_runtime": 289.51256942749023, "_step": 12, "eval/1_eval": {"_type": "video-file", "sha256": "f136ddfa6a75f8c38319eb69e397cb0cf05992c189494f06c72a123d44e57cad", "size": 1169127, "path": "media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4"}, "eval/num_success": 6.0, "episode": 6.0, "eval/success_rate": 1.0, "eval/success_rate_lb_0.01": 0.8098405599594116, "eval/success_rate_ub_0.01": 1.0, "eval/success_rate_lb_0.025": 0.780625581741333, "eval/success_rate_ub_0.025": 1.0, "eval/success_rate_lb_0.05": 0.6978645324707031, "eval/success_rate_ub_0.05": 1.0, "eval/success_rate_lb_0.1": 0.7496918439865112, "eval/success_rate_ub_0.1": 1.0, "eval/2_eval": {"_type": "video-file", "sha256": "9019146f9223df893e271a59c4c8538311ba44dc2fd24a72c9dd4b2c64812680", "size": 1170383, "path": "media/videos/eval/2_eval_2_9019146f9223df893e27.mp4"}, "eval/3_eval": {"_type": "video-file", "sha256": "a9352300f4a1f2d4fc59db4120a6eeb7a174fd0b71713a6348112847f63b66a3", "size": 1198800, "path": "media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4"}, "eval/4_eval": {"_type": "video-file", "sha256": "ff051e1686bf761b709a2c0ca3d274928506972c8da93aa7332f58ee411cb53a", "size": 1236656, "path": "media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4"}, "eval/5_eval": {"_type": "video-file", "sha256": "1a0fa497056b618ea0c00d6fc66b9c6ec244aca7406d79762d07999bb1f21f05", "size": 1158233, "path": "media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4"}, "eval/6_eval": {"_type": "video-file", "sha256": "a90b503bf7614a8b51c57e646cac1e6c87e69c6f37fa340d991743bbf1419c98", "size": 1334222, "path": "media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4"}}']}, 'wandb-history.jsonl': {'offset': 12, 'content': ['{"global_step": 796, "_timestamp": 1737252513.5501094, "eval/num_success": 6.0, "episode": 6.0, "eval/success_rate": 1.0, "eval/success_rate_lb_0.01": 0.8098405599594116, "eval/success_rate_ub_0.01": 1.0, "eval/success_rate_lb_0.025": 0.780625581741333, "eval/success_rate_ub_0.025": 1.0, "eval/success_rate_lb_0.05": 0.6978645324707031, "eval/success_rate_ub_0.05": 1.0, "eval/success_rate_lb_0.1": 0.7496918439865112, "eval/success_rate_ub_0.1": 1.0, "_runtime": 289.51256942749023, "_step": 12}']}, 'wandb-events.jsonl': {'offset': 9, 'content': ['{"system.proc.memory.availableMB": 54499.66, "system.memory": 15.13, "system.proc.memory.rssMB": 940.93, "system.proc.memory.percent": 1.47, "system.cpu": 13.47, "system.cpu.0.cpu_percent": 23.32, "system.cpu.1.cpu_percent": 23.09, "system.cpu.2.cpu_percent": 49.47, "system.cpu.3.cpu_percent": 26.25, "system.cpu.4.cpu_percent": 27.35, "system.cpu.5.cpu_percent": 24.99, "system.cpu.6.cpu_percent": 24.06, "system.cpu.7.cpu_percent": 24.03, "system.proc.cpu.threads": 59, "system.network.sent": 41607873249.27, "system.network.recv": 28071465410.33, "system.disk.\\\\.usagePercent": 76.67, "system.disk.\\\\.usageGB": 665.91, "system.disk.in": 0.27, "system.disk.out": 7344.52, "system.gpu.0.memoryAllocated": 28.37, "system.gpu.0.memoryAllocatedBytes": 2437205196.8, "system.gpu.0.memory": 4.0, "system.gpu.0.gpu": 13.13, "system.gpu.0.temp": 47.73, "system.gpu.0.powerWatts": 49.35, "system.gpu.0.powerPercent": 22.63, "_wandb": true, "_timestamp": 1737252554.561908, "_runtime": 330.524368}', '{"system.proc.memory.availableMB": 54431.22, "system.memory": 15.25, "system.proc.memory.rssMB": 933.58, "system.proc.memory.percent": 1.45, "system.cpu": 7.43, "system.cpu.0.cpu_percent": 31.45, "system.cpu.1.cpu_percent": 29.58, "system.cpu.2.cpu_percent": 47.41, "system.cpu.3.cpu_percent": 30.83, "system.cpu.4.cpu_percent": 27.8, "system.cpu.5.cpu_percent": 29.28, "system.cpu.6.cpu_percent": 26.15, "system.cpu.7.cpu_percent": 23.95, "system.proc.cpu.threads": 56, "system.network.sent": 46013138381.0, "system.network.recv": 31047672563.0, "system.disk.\\\\.usagePercent": 76.75, "system.disk.\\\\.usageGB": 666.75, "system.disk.in": 0.29, "system.disk.out": 8135.24, "system.gpu.0.memoryAllocated": 28.37, "system.gpu.0.memoryAllocatedBytes": 2436934314.67, "system.gpu.0.memory": 8.73, "system.gpu.0.gpu": 31.4, "system.gpu.0.temp": 49.8, "system.gpu.0.powerWatts": 57.08, "system.gpu.0.powerPercent": 26.55, "_wandb": true, "_timestamp": 1737252584.56238, "_runtime": 360.52484}']}, 'output.log': {'offset': 327, 'content': ['2025-01-19T02:09:11.352145 [INFO] [1737252549.414495]: Storing episode...\n', '2025-01-19T02:09:11.352145 [rosbag_record-21] killing on exit\n', '2025-01-19T02:09:11.352145 [print_text-20] killing on exit\n', '2025-01-19T02:09:11.352145 [pub_text-19] killing on exit\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.184000]: Stored episode 7.\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.186327]: Saving video...\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.399846]: Video saved!\n', '2025-01-19T02:09:11.352145 Starting episode 7\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.493449]: resetting environment\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.525712]: cleared current plan\n', '2025-01-19T02:09:11.352145 [INFO] [1737252550.533201]: moving to home\n', '2025-01-19T02:09:15.355592 [INFO] [1737252554.146813]: reached home\n', '2025-01-19T02:09:15.355592 [INFO] [1737252554.166841]: reset action history\n', '2025-01-19T02:09:19.358922 [INFO] [1737252557.025628]: environment reset\n', '2025-01-19T02:09:19.358922 Press Enter to continue... after reseting env. To rate prev episode, press 0 for failure and 1 for success\n', '2025-01-19T02:09:23.364233 proceeding to start episode!\n', '2025-01-19T02:09:23.364233 Comuting lower confidence bound\n', '2025-01-19T02:09:23.364233 Comuting upper confidence bound\n', '2025-01-19T02:09:23.364233 Comuting lower confidence bound\n', '2025-01-19T02:09:23.364233 Comuting upper confidence bound\n', '2025-01-19T02:09:23.364233 Comuting lower confidence bound\n', '2025-01-19T02:09:23.364233 Comuting upper confidence bound\n', '2025-01-19T02:09:23.364233 Comuting lower confidence bound\n', '2025-01-19T02:09:23.364233 Comuting upper confidence bound\n', '2025-01-19T02:09:23.364233 [INFO] [1737252561.894930]: resetting environment\n', '2025-01-19T02:09:23.364233 [INFO] [1737252561.926678]: cleared current plan\n', '2025-01-19T02:09:23.364233 [INFO] [1737252561.928515]: moving to home\n', '2025-01-19T02:09:25.365484 [INFO] [1737252562.937209]: reached home\n', '2025-01-19T02:09:25.365484 [INFO] [1737252562.946625]: reset action history\n', '2025-01-19T02:09:27.367948 [INFO] [1737252565.826750]: environment reset\n', '2025-01-19T02:09:27.367948 bag_path_name:=/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/episode_rosbags/episode_7_2025-01-18-21-09-25.bag\n', '2025-01-19T02:09:27.367948 ... logging to /home/leonmkim/.ros/log/1b77c894-d3af-11ef-96d2-5defea869005/roslaunch-leonmkim-ROG-Strix-G15CS-G15CS-2577081.log\n', '2025-01-19T02:09:29.370300 \x1b[1mstarted roslaunch server http://158.130.50.37:38859/\n', '2025-01-19T02:09:29.370300 SUMMARY\n', '2025-01-19T02:09:29.370300 ========\n', '2025-01-19T02:09:29.370300 PARAMETERS\n', '2025-01-19T02:09:29.370300  * /rosdistro: noetic\n', '2025-01-19T02:09:29.370300  * /rosversion: 1.16.0\n', '2025-01-19T02:09:29.370300 NODES\n', '2025-01-19T02:09:29.370300   /\n', '2025-01-19T02:09:29.370300     print_text (rostopic/rostopic)\n', '2025-01-19T02:09:29.370300     pub_text (rostopic/rostopic)\n', '2025-01-19T02:09:29.370300     rosbag_record (rosbag/record)\n', '2025-01-19T02:09:29.370300 \x1b[1mROS_MASTER_URI=http://localhost:11311\n', '2025-01-19T02:09:29.370300 \x1b[1mprocess[pub_text-22]: started with pid [2578397]\n', '2025-01-19T02:09:29.370300 \x1b[1mprocess[print_text-23]: started with pid [2578398]\n', '2025-01-19T02:09:29.370300 \x1b[1mprocess[rosbag_record-24]: started with pid [2578399]\n', '2025-01-19T02:09:29.370300 started bagging!\n']}}, 'dropped': 0}}
2025-01-18 21:09:44,930 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:09:44,930 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:09:45,077 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:45,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:46,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:47,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:48,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:49,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:50,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:50,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:51,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:52,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:53,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:54,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:55,117 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:55,175 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:09:55,176 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4 with policy now
2025-01-18 21:09:55,217 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:09:55,217 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:09:55,218 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:09:55,218 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:09:55,471 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/ob9870et-media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4
2025-01-18 21:09:55,624 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:55,624 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:09:55,624 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4
2025-01-18 21:09:55,625 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:09:55,919 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/kqy5bfoe-media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4
2025-01-18 21:09:56,116 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:56,258 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:09:57,111 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:57,625 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:58,112 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:59,116 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:09:59,626 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:09:59,949 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:09:59,949 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:10:00,117 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:01,117 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:01,794 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:02,109 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:03,114 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:03,627 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:04,153 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:04,429 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:10:04,430 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:10:04,430 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:10:04,627 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:10:04,628 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:10:05,115 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:05,628 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:06,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:07,138 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:07,317 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:07,628 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:08,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:09,143 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:09,629 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:10,125 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:11,132 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:12,133 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:12,970 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:13,144 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:14,127 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:14,563 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:10:14,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:10:14,929 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:10:15,127 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:15,632 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:16,128 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:17,128 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:17,633 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:18,105 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:18,128 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:19,128 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:19,633 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:20,128 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:21,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:22,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:23,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:23,956 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:24,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:25,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:26,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:27,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:27,636 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:28,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:29,129 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:29,637 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:29,929 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:10:29,929 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:10:30,045 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:30,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:31,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:32,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:33,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:33,639 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:34,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:35,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:35,639 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:35,956 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:36,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:37,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:38,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:39,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:40,131 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:41,140 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:41,497 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:41,642 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:41,827 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:10:41,827 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_4562dd864115606b6291.mp4 with policy now
2025-01-18 21:10:41,837 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:10:41,838 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:10:41,838 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:10:41,838 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:10:42,116 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/cwk3e2tk-media/videos/eval/8_eval_8_4562dd864115606b6291.mp4
2025-01-18 21:10:42,144 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:42,642 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:10:42,642 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/8_eval_8_4562dd864115606b6291.mp4
2025-01-18 21:10:42,642 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:10:42,958 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/2hglpkjk-media/videos/eval/8_eval_8_4562dd864115606b6291.mp4
2025-01-18 21:10:43,150 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:43,642 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:44,167 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:44,563 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:10:44,982 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:10:44,982 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:10:45,154 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:46,165 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:46,553 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:47,211 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:47,644 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:48,173 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:49,178 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:49,487 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:10:49,487 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:10:49,492 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:10:49,645 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:49,645 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:10:49,645 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:10:49,903 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/z2qrvsct-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:10:50,167 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:51,171 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:51,646 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:51,982 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:52,189 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:53,173 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:53,646 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:54,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:55,180 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:56,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:57,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:57,974 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:10:58,199 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:59,192 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:10:59,649 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:10:59,935 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:10:59,935 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:11:00,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:01,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:01,650 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:02,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:03,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:03,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:04,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:05,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:05,652 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:06,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:07,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:07,652 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:08,194 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:08,245 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:09,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:10,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:11,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:12,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:13,195 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:13,961 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:14,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:14,564 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:11:14,935 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:11:14,936 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:11:15,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:16,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:17,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:18,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:19,095 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:19,196 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:20,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:21,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:21,657 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:22,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:23,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:24,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:24,458 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:25,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:26,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:27,221 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:27,660 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:27,811 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:11:27,811 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4 with policy now
2025-01-18 21:11:27,814 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:11:27,814 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:11:27,814 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:11:27,814 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:11:28,082 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/0391rdg1-media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4
2025-01-18 21:11:28,209 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:28,660 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:11:28,661 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4
2025-01-18 21:11:28,661 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:11:28,977 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/qrkurpbu-media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4
2025-01-18 21:11:29,214 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:29,661 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:29,846 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:29,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:11:29,959 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:11:30,217 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:31,214 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:32,224 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:33,230 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:33,662 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:34,260 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:34,962 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:35,225 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:35,663 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:36,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:37,221 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:37,544 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:11:37,544 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:11:37,545 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:11:37,664 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:37,664 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:11:37,664 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:11:38,261 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:39,242 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:39,665 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:39,967 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:40,244 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:41,233 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:41,665 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:42,239 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:43,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:43,666 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:11:44,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:44,564 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:11:44,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:11:44,940 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:11:45,063 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:45,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:46,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:47,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:48,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:49,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:50,234 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:50,964 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:51,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:52,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:53,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:54,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:55,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:55,965 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:11:56,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:57,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:58,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:59,236 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:11:59,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:11:59,941 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:12:00,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:01,084 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:01,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:02,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:03,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:04,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:05,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:06,238 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:06,965 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:07,238 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:08,238 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:09,238 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:10,246 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:10,269 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:12:10,269 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4 with policy now
2025-01-18 21:12:10,313 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:12:10,313 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:12:10,314 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:12:10,314 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:12:10,535 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/2n9jc6ae-media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4
2025-01-18 21:12:10,677 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:12:10,677 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4
2025-01-18 21:12:10,678 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:12:10,979 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/i301d7bv-media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4
2025-01-18 21:12:11,257 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:11,678 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:12,277 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:12,339 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:13,291 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:14,264 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:14,565 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:12:14,953 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:12:14,954 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:12:15,267 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:15,679 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:16,271 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:17,265 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:17,680 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:18,097 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:18,270 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:19,274 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:20,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:21,302 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:22,305 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:22,600 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:12:22,600 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:12:22,601 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:12:22,682 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:12:22,682 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:12:22,953 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/szyye1o9-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:12:23,284 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:23,566 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:23,682 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:24,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:25,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:25,683 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:26,299 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:27,299 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:27,684 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:28,314 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:28,705 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:29,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:29,971 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:12:29,972 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:12:30,308 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:31,309 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:32,299 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:33,297 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:34,066 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:34,300 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:35,293 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:35,687 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:36,294 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:37,294 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:37,688 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:38,294 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:39,130 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:39,294 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:40,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:41,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:42,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:43,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:44,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:44,517 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:44,565 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:12:44,945 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:12:44,945 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:12:45,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:45,691 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:46,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:47,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:48,295 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:49,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:49,988 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:50,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:51,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:52,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:53,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:54,296 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:54,988 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:12:55,298 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:56,307 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:56,448 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:12:56,448 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4 with policy now
2025-01-18 21:12:56,451 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:12:56,451 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:12:56,451 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:12:56,452 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:12:56,696 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:12:56,696 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4
2025-01-18 21:12:56,696 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:12:56,718 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/nyybbuco-media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4
2025-01-18 21:12:57,015 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/ykakra7a-media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4
2025-01-18 21:12:57,305 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:57,696 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:12:58,346 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:59,359 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:12:59,966 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:12:59,967 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:13:00,083 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:00,335 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:01,356 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:01,698 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:02,361 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:03,381 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:04,368 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:04,652 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:13:04,652 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:13:04,653 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:13:04,699 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:13:04,699 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:13:05,395 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:05,442 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:05,699 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:06,382 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:07,390 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:07,700 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:08,400 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:09,407 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:09,701 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:10,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:11,001 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:11,406 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:12,428 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:13,433 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:14,429 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:14,566 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:13:14,988 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:13:14,988 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:13:15,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:16,105 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:16,420 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:17,461 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:17,704 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:18,420 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:19,420 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:19,705 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:20,420 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:21,420 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:21,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:22,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:23,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:24,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:25,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:26,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:26,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:27,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:28,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:29,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:29,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:13:29,960 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:13:30,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:31,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:32,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:32,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:33,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:34,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:35,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:36,422 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:37,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:37,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:38,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:39,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:40,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:41,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:42,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:42,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:43,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:44,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:44,566 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:13:44,974 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:13:44,975 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:13:45,419 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:13:45,419 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4 with policy now
2025-01-18 21:13:45,439 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:45,444 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:13:45,444 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:13:45,444 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:13:45,445 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:13:45,702 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/qc4ilx14-media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4
2025-01-18 21:13:45,715 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:45,715 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:13:45,715 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4
2025-01-18 21:13:45,716 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:13:46,020 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/0ea2c26n-media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4
2025-01-18 21:13:46,445 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:47,438 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:47,716 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:48,467 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:48,507 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:49,499 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:49,717 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:50,457 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:51,455 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:52,439 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:53,472 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:53,718 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:13:53,909 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:54,458 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:55,468 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:56,457 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:57,470 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:58,475 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:59,017 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:13:59,477 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:13:59,998 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:13:59,998 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:14:00,477 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:01,482 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:02,515 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:03,505 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:03,722 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:14:03,722 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:14:03,723 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:14:03,723 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:14:03,724 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:14:04,524 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:04,724 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:14:04,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:05,492 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:05,724 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:06,515 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:07,497 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:08,523 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:09,548 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:09,726 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:09,837 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:10,525 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:11,551 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:12,522 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:13,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:14,545 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:14,567 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:14:14,971 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:14:14,972 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:14:15,126 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:15,561 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:16,571 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:17,538 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:17,729 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:18,538 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:19,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:19,730 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:20,538 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:21,060 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:21,538 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:21,731 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:22,538 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:23,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:24,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:25,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:26,061 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:26,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:27,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:28,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:29,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:29,963 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:14:29,963 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:14:30,539 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:31,098 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:31,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:32,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:33,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:34,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:35,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:36,540 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:37,061 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:37,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:38,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:39,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:40,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:41,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:42,061 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:42,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:43,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:44,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:44,567 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:14:44,963 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:14:44,963 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:14:45,577 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:45,741 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:46,154 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:14:46,154 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4 with policy now
2025-01-18 21:14:46,157 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:14:46,157 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:14:46,157 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:14:46,158 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:14:46,444 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/w5wwaed3-media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4
2025-01-18 21:14:46,555 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:46,741 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:14:46,741 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4
2025-01-18 21:14:46,742 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:14:47,056 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/wnenej0g-media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4
2025-01-18 21:14:47,206 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:47,588 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:47,742 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:48,615 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:49,587 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:50,593 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:51,609 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:51,743 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:52,600 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:52,965 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:53,602 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:53,744 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:14:54,608 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:55,630 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:56,607 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:57,624 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:58,081 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:14:58,609 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:59,643 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:14:59,969 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:14:59,969 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:15:00,621 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:01,633 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:02,636 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:03,096 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:03,652 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:04,670 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:05,672 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:06,649 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:07,657 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:08,658 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:09,073 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:09,672 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:10,665 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:11,675 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:12,666 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:13,665 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:13,752 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:13,752 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:15:13,808 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:15:13,809 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:15:13,809 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:15:14,388 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:14,436 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/yyx1h8uq-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:15:14,567 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:15:14,692 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:14,753 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:15:14,978 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:15:14,978 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:15:15,723 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:15,753 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:16,690 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:17,688 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:18,720 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:19,569 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:19,713 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:19,755 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:20,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:21,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:22,715 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:23,727 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:24,735 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:25,116 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:25,761 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:26,801 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:27,777 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:28,728 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:29,728 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:29,759 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:29,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:15:29,976 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:15:30,728 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:31,102 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:31,728 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:31,760 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:32,728 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:33,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:34,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:35,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:36,102 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:36,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:37,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:38,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:39,732 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:40,729 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:41,103 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:41,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:42,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:43,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:44,568 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:15:44,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:44,976 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:15:44,976 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:15:45,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:46,118 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:46,730 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:47,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:48,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:49,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:50,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:51,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:52,103 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:52,731 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:53,732 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:54,733 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:55,733 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:56,752 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:57,197 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:15:57,363 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:15:57,363 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4 with policy now
2025-01-18 21:15:57,377 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:15:57,378 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:15:57,378 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:15:57,378 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:15:57,643 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/p9egipbp-media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4
2025-01-18 21:15:57,739 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:57,770 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:57,770 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:15:57,771 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4
2025-01-18 21:15:57,771 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:15:58,066 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/pnz7oe18-media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4
2025-01-18 21:15:58,762 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:59,752 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:15:59,771 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:15:59,991 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:15:59,992 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:16:00,755 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:01,772 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:01,778 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:02,757 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:03,009 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:03,769 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:04,779 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:05,774 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:05,819 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:06,808 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:07,774 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:07,775 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:16:07,785 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:07,880 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:16:07,880 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:16:07,881 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:16:08,357 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:08,775 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:16:08,835 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:09,775 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:09,783 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:10,820 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:11,783 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:12,790 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:13,737 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:13,779 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:13,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:14,568 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:16:14,822 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:14,981 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:16:14,982 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:16:15,810 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:16,841 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:17,837 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:18,824 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:18,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:19,781 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:19,810 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:20,810 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:21,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:22,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:23,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:24,122 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:24,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:25,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:26,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:27,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:28,811 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:29,122 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:29,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:29,979 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:16:29,979 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:16:30,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:31,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:32,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:33,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:34,123 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:34,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:35,812 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:36,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:37,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:38,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:39,123 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:39,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:40,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:41,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:42,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:43,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:44,569 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:16:44,570 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:44,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:44,979 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:16:44,979 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:16:45,813 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:46,831 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:47,097 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:16:47,097 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4 with policy now
2025-01-18 21:16:47,117 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:16:47,117 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:16:47,117 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:16:47,118 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:16:47,368 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/psj3ohm0-media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4
2025-01-18 21:16:47,797 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:47,797 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:16:47,797 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4
2025-01-18 21:16:47,797 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:16:47,864 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:48,117 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/y5yntkla-media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4
2025-01-18 21:16:48,854 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:49,849 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:50,127 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:50,905 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:51,798 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:51,851 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:52,854 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:53,848 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:54,847 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:55,139 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:16:55,860 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:56,800 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:16:56,884 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:57,877 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:58,903 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:16:59,903 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:00,029 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:17:00,029 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:17:00,902 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:01,138 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:01,941 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:02,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:03,803 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:17:03,948 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:03,948 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:17:03,948 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:17:03,948 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:17:04,803 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:04,804 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:17:04,907 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:05,935 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:06,804 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:06,936 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:07,138 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:07,923 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:08,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:09,935 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:10,806 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:10,948 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:11,933 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:12,167 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:12,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:13,969 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:14,569 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:17:14,998 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:17:14,998 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:17:15,045 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:16,010 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:16,952 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:17,541 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:17,963 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:18,809 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:18,956 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:19,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:20,810 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:20,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:21,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:22,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:23,103 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:23,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:24,812 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:24,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:25,957 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:26,813 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:26,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:27,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:28,545 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:28,814 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:28,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:29,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:29,985 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:17:29,985 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:17:30,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:31,958 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:32,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:33,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:34,131 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:34,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:35,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:36,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:37,959 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:38,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:39,131 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:39,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:40,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:41,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:42,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:43,960 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:44,131 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:44,570 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:17:44,993 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:45,004 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:17:45,004 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:17:45,975 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:45,998 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:17:45,998 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4 with policy now
2025-01-18 21:17:46,041 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:17:46,042 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:17:46,042 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:17:46,046 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:17:46,296 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/05pnzh1z-media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4
2025-01-18 21:17:46,822 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:46,822 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:17:46,822 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4
2025-01-18 21:17:46,822 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:17:46,973 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:47,145 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/xnrnd6np-media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4
2025-01-18 21:17:47,822 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:47,997 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:48,823 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:49,004 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:49,734 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:49,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:50,984 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:52,026 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:52,824 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:53,002 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:54,027 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:54,825 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:17:55,043 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:55,172 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:17:56,071 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:57,033 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:58,046 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:59,046 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:17:59,994 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:17:59,994 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:18:00,055 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:00,421 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:00,828 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:18:01,021 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:18:01,021 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:18:01,021 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:18:01,022 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:01,089 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/tncyld2n-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:18:01,828 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:18:02,024 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:02,829 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:03,067 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:04,030 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:04,829 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:05,044 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:06,078 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:06,184 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:06,830 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:07,054 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:08,059 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:09,077 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:10,054 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:11,081 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:12,076 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:12,182 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:13,084 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:14,073 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:14,570 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:18:15,067 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:18:15,067 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:18:15,109 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:16,147 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:17,113 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:18,107 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:18,179 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:19,089 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:20,089 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:20,836 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:21,089 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:22,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:22,838 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:23,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:24,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:24,152 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:25,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:26,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:27,090 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:28,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:29,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:29,152 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:29,988 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:18:29,988 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:18:30,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:31,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:32,091 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:33,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:34,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:34,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:35,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:36,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:37,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:38,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:39,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:40,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:40,151 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:41,093 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:42,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:43,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:44,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:44,571 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:18:44,989 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:18:44,989 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:18:45,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:45,152 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:46,094 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:47,114 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:47,603 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:18:47,603 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4 with policy now
2025-01-18 21:18:47,609 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:18:47,610 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:18:47,610 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:18:47,610 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:18:47,850 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:18:47,850 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4
2025-01-18 21:18:47,850 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:18:47,933 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/p7cj1ipe-media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4
2025-01-18 21:18:48,120 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:48,231 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/yi5afswe-media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4
2025-01-18 21:18:48,850 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:49,127 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:50,152 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:50,155 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:50,851 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:51,177 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:52,149 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:52,852 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:53,184 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:54,171 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:55,157 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:18:55,158 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:56,188 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:56,854 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:18:57,186 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:58,199 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:18:59,226 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:00,012 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:19:00,012 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:19:00,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:00,216 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:01,184 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:02,186 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:03,235 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:04,198 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:05,208 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:06,180 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:06,237 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:06,858 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:19:07,099 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:19:07,099 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:19:07,100 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:19:07,208 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:07,858 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:19:07,859 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:19:08,215 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:08,859 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:09,243 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:10,229 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:10,860 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:11,223 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:11,428 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:12,232 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:12,861 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:13,235 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:14,267 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:14,577 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:19:15,059 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:19:15,059 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:19:15,283 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:16,268 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:17,172 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:17,259 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:18,269 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:19,260 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:20,287 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:21,262 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:22,191 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:22,330 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:23,308 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:24,274 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:24,866 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:25,274 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:26,274 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:26,867 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:27,277 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:28,180 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:28,275 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:29,275 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:29,995 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:19:29,995 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:19:30,275 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:31,276 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:32,276 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:33,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:33,276 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:34,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:35,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:36,279 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:37,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:38,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:39,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:39,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:40,281 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:41,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:42,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:43,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:44,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:44,280 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:44,571 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:19:44,999 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:19:44,999 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:19:45,281 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:46,282 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:47,285 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:48,281 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:49,181 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:49,289 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:50,285 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:51,286 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:52,326 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:53,090 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:19:53,090 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4 with policy now
2025-01-18 21:19:53,101 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:19:53,101 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:19:53,101 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:19:53,103 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:19:53,328 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:53,371 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/cy0ojnxt-media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4
2025-01-18 21:19:53,879 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:53,879 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:19:53,879 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4
2025-01-18 21:19:53,879 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:19:54,239 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:19:54,245 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/s7eojymz-media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4
2025-01-18 21:19:54,343 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:54,879 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:55,331 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:56,344 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:57,326 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:58,343 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:58,881 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:19:59,307 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:19:59,913 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:00,036 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:20:00,036 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:20:00,306 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:01,330 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:02,342 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:02,883 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:03,381 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:04,369 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:05,198 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:05,356 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:06,396 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:07,436 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:07,885 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:20:08,171 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:20:08,171 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:20:08,172 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:20:08,381 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:08,886 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:08,886 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:20:08,886 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:20:09,382 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:10,247 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:10,395 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:10,886 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:11,386 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:12,389 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:13,415 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:14,410 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:14,572 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:20:14,888 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:15,020 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:20:15,020 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:20:15,251 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:15,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:16,404 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:17,443 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:18,416 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:19,416 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:20,449 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:21,215 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:21,448 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:22,442 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:23,447 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:24,445 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:25,434 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:26,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:26,893 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:26,915 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:27,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:28,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:28,894 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:29,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:30,011 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:20:30,012 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:20:30,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:31,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:32,212 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:32,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:33,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:34,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:35,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:36,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:37,424 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:38,213 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:38,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:39,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:40,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:41,425 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:42,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:42,901 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:43,293 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:43,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:44,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:44,572 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:20:45,011 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:20:45,012 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:20:45,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:46,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:47,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:48,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:49,213 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:49,426 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:50,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:51,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:52,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:53,486 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:53,965 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:20:53,966 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4 with policy now
2025-01-18 21:20:53,987 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:20:53,988 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:20:53,988 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:20:53,988 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:20:54,277 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/g1t2eo87-media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4
2025-01-18 21:20:54,438 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:54,906 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:54,907 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:20:54,907 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4
2025-01-18 21:20:54,907 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval
2025-01-18 21:20:55,026 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:20:55,220 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/pvmpldhg-media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4
2025-01-18 21:20:55,478 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:55,907 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:56,467 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:56,907 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:57,496 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:58,489 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:20:58,908 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:20:59,493 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:00,020 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:21:00,020 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:21:00,115 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:00,517 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:00,909 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:01,518 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:02,512 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:02,910 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:03,531 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:04,548 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:05,285 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:05,544 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:06,554 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:07,553 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:08,585 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:09,243 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:21:09,244 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:09,244 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:09,613 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:09,913 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:09,913 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:10,609 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:10,908 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-18 21:21:10,908 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: log_artifact
2025-01-18 21:21:10,914 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:11,137 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:21:11,342 INFO    wandb-upload_2:2577265 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp2x7lu8zi
2025-01-18 21:21:11,593 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:11,744 INFO    SenderThread:2577265 [sender.py:send_request_log_artifact():1525] logged artifact run-2t6elatx-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1MjA0NTYxNQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MzExNTAzNA==', 'latestArtifact': None}}
2025-01-18 21:21:11,744 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:21:11,744 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json with policy now
2025-01-18 21:21:11,745 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:11,745 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:21:11,745 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:11,746 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:11,914 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:11,914 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json
2025-01-18 21:21:11,914 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media
2025-01-18 21:21:11,915 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table
2025-01-18 21:21:11,915 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval
2025-01-18 21:21:11,940 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/hzqzkb5b-media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json
2025-01-18 21:21:12,206 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/e7jjem0a-media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json
2025-01-18 21:21:12,593 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:13,594 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:14,573 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:21:14,594 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:15,016 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: stop_status
2025-01-18 21:21:15,016 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: stop_status
2025-01-18 21:21:15,595 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:16,596 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:17,233 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:17,595 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:18,596 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:19,596 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:20,596 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:21,596 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:22,597 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:22,893 DEBUG   SenderThread:2577265 [sender.py:send():379] send: telemetry
2025-01-18 21:21:22,894 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:22,919 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:22,941 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,941 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,941 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,942 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,942 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,942 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,943 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,943 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,943 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,944 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,944 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,944 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,945 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,945 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,945 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,946 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,946 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,946 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,946 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,947 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:22,947 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:22,947 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:23,034 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: log_artifact
2025-01-18 21:21:23,034 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: log_artifact
2025-01-18 21:21:23,044 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: partial_history
2025-01-18 21:21:23,188 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/ahcq43fd-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:23,382 INFO    wandb-upload_2:2577265 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpcblxnubg
2025-01-18 21:21:23,597 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:23,800 INFO    SenderThread:2577265 [sender.py:send_request_log_artifact():1525] logged artifact run-2t6elatx-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1MjA0NTczOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MzExNTA1Nw==', 'latestArtifact': None}}
2025-01-18 21:21:23,800 DEBUG   SenderThread:2577265 [sender.py:send():379] send: files
2025-01-18 21:21:23,800 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json with policy now
2025-01-18 21:21:23,800 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:21:23,800 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:23,801 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:23,920 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:23,921 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:23,921 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json
2025-01-18 21:21:23,921 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval
2025-01-18 21:21:23,999 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/mr0gm6ct-media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json
2025-01-18 21:21:24,280 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/d6rv1i6i-media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json
2025-01-18 21:21:24,598 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:24,921 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:25,597 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:26,597 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:27,598 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:28,233 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:28,599 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:29,597 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:29,689 DEBUG   SenderThread:2577265 [sender.py:send():379] send: exit
2025-01-18 21:21:29,689 INFO    SenderThread:2577265 [sender.py:send_exit():586] handling exit code: 0
2025-01-18 21:21:29,690 INFO    SenderThread:2577265 [sender.py:send_exit():588] handling runtime: 1065
2025-01-18 21:21:29,691 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:29,691 INFO    SenderThread:2577265 [sender.py:send_exit():594] send defer
2025-01-18 21:21:29,691 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:29,691 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 0
2025-01-18 21:21:29,691 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:29,691 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 0
2025-01-18 21:21:29,691 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 1
2025-01-18 21:21:29,691 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:29,691 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 1
2025-01-18 21:21:29,692 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:29,692 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 1
2025-01-18 21:21:29,692 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 2
2025-01-18 21:21:29,692 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:29,692 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 2
2025-01-18 21:21:29,692 INFO    HandlerThread:2577265 [system_monitor.py:finish():203] Stopping system monitor
2025-01-18 21:21:29,692 DEBUG   SystemMonitor:2577265 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2025-01-18 21:21:29,692 DEBUG   SystemMonitor:2577265 [system_monitor.py:_start():183] Publishing last batch of metrics
2025-01-18 21:21:29,693 INFO    HandlerThread:2577265 [interfaces.py:finish():200] Joined cpu monitor
2025-01-18 21:21:29,693 INFO    HandlerThread:2577265 [interfaces.py:finish():200] Joined disk monitor
2025-01-18 21:21:29,744 INFO    HandlerThread:2577265 [interfaces.py:finish():200] Joined gpu monitor
2025-01-18 21:21:29,744 INFO    HandlerThread:2577265 [interfaces.py:finish():200] Joined memory monitor
2025-01-18 21:21:29,744 INFO    HandlerThread:2577265 [interfaces.py:finish():200] Joined network monitor
2025-01-18 21:21:29,744 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:29,744 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 2
2025-01-18 21:21:29,744 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 3
2025-01-18 21:21:29,745 DEBUG   SenderThread:2577265 [sender.py:send():379] send: stats
2025-01-18 21:21:29,745 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:29,745 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 3
2025-01-18 21:21:29,745 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:29,745 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 3
2025-01-18 21:21:29,745 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 4
2025-01-18 21:21:29,745 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:29,745 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 4
2025-01-18 21:21:29,923 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:30,924 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:36,270 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-18 21:21:36,270 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:36,270 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: keepalive
2025-01-18 21:21:36,271 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:36,271 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 4
2025-01-18 21:21:36,271 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 5
2025-01-18 21:21:36,271 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: poll_exit
2025-01-18 21:21:36,271 DEBUG   SenderThread:2577265 [sender.py:send():379] send: history
2025-01-18 21:21:36,271 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: summary_record
2025-01-18 21:21:36,272 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:36,272 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 5
2025-01-18 21:21:36,272 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:36,272 DEBUG   SenderThread:2577265 [sender.py:send():379] send: summary
2025-01-18 21:21:36,276 INFO    SenderThread:2577265 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end
2025-01-18 21:21:36,277 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:36,277 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 5
2025-01-18 21:21:36,277 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 6
2025-01-18 21:21:36,277 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:36,277 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 6
2025-01-18 21:21:36,277 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:36,277 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 6
2025-01-18 21:21:36,283 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: status_report
2025-01-18 21:21:36,426 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 7
2025-01-18 21:21:36,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:36,427 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 7
2025-01-18 21:21:36,427 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:36,427 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 7
2025-01-18 21:21:36,691 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-18 21:21:36,927 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/config.yaml
2025-01-18 21:21:36,927 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:39,909 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 8
2025-01-18 21:21:39,909 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: poll_exit
2025-01-18 21:21:39,910 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:39,910 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 8
2025-01-18 21:21:39,910 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:39,910 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 8
2025-01-18 21:21:39,910 INFO    SenderThread:2577265 [job_builder.py:build():440] Attempting to build job artifact
2025-01-18 21:21:39,913 INFO    SenderThread:2577265 [job_builder.py:_get_source_type():569] is repo sourced job
2025-01-18 21:21:39,915 INFO    SenderThread:2577265 [job_builder.py:build():545] adding wandb-job metadata file
2025-01-18 21:21:39,928 INFO    Thread-12 :2577265 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:39,934 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 9
2025-01-18 21:21:39,934 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:39,934 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 9
2025-01-18 21:21:39,935 DEBUG   SenderThread:2577265 [sender.py:send():379] send: artifact
2025-01-18 21:21:40,322 INFO    wandb-upload_1:2577265 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpe1wv_oo5
2025-01-18 21:21:40,322 INFO    wandb-upload_2:2577265 [upload_job.py:push():85] Skipped uploading /tmp/tmpfj0s81cv/wandb-job.json
2025-01-18 21:21:40,506 INFO    wandb-upload_3:2577265 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp8gzxv2_4
2025-01-18 21:21:40,692 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-18 21:21:41,186 INFO    SenderThread:2577265 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1MjA0NTk0MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1MjAyMTE2MQ==', 'versionIndex': 450}}}
2025-01-18 21:21:41,186 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:41,186 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 9
2025-01-18 21:21:41,186 INFO    SenderThread:2577265 [dir_watcher.py:finish():358] shutting down directory watcher
2025-01-18 21:21:41,929 INFO    SenderThread:2577265 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files
2025-01-18 21:21:41,929 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/config.yaml config.yaml
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json wandb-summary.json
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0 events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/requirements.txt requirements.txt
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-metadata.json wandb-metadata.json
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch
2025-01-18 21:21:41,930 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log output.log
2025-01-18 21:21:41,932 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/diff.patch diff.patch
2025-01-18 21:21:41,933 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/code/FISH/eval_robot.py code/FISH/eval_robot.py
2025-01-18 21:21:41,936 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4 media/videos/eval/17_eval_17_cb16d97a7d17e520d767.mp4
2025-01-18 21:21:41,936 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4 media/videos/eval/16_eval_16_dec0060310bb185ed9e7.mp4
2025-01-18 21:21:41,936 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4 media/videos/eval/13_eval_13_f717f47f7f63a8b77820.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4 media/videos/eval/0_eval_0_373935c9a53d50a21516.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/2_eval_2_9019146f9223df893e27.mp4 media/videos/eval/2_eval_2_9019146f9223df893e27.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4 media/videos/eval/1_eval_1_f136ddfa6a75f8c38319.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4 media/videos/eval/15_eval_15_7379e8bc33b5c38d196c.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4 media/videos/eval/9_eval_9_0cb54919d1152aace357.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/8_eval_8_4562dd864115606b6291.mp4 media/videos/eval/8_eval_8_4562dd864115606b6291.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4 media/videos/eval/10_eval_10_194dcd6049c4233da56c.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4 media/videos/eval/14_eval_14_6cb31531142c32910aad.mp4
2025-01-18 21:21:41,938 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4 media/videos/eval/12_eval_12_8a945bf8da73837c5024.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4 media/videos/eval/7_eval_7_d9b2fef5e246e9ba0a8b.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4 media/videos/eval/6_eval_6_a90b503bf7614a8b51c5.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4 media/videos/eval/19_eval_19_0ae4f8c9bd51774c0b89.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4 media/videos/eval/18_eval_18_c883e9e40544a093c095.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4 media/videos/eval/11_eval_11_136a95e750557f39dd09.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4 media/videos/eval/4_eval_4_ff051e1686bf761b709a.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4 media/videos/eval/5_eval_5_1a0fa497056b618ea0c0.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4 media/videos/eval/3_eval_3_a9352300f4a1f2d4fc59.mp4
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json media/table/eval/total_success_rate_ci_21_984e345eb950dc10e24c.table.json
2025-01-18 21:21:41,939 INFO    SenderThread:2577265 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json media/table/eval/success_rate_ci_20_046a987f9352685cc2d7.table.json
2025-01-18 21:21:41,940 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 10
2025-01-18 21:21:41,940 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: poll_exit
2025-01-18 21:21:41,940 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:41,940 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 10
2025-01-18 21:21:41,940 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:41,940 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 10
2025-01-18 21:21:41,940 INFO    SenderThread:2577265 [file_pusher.py:finish():169] shutting down file pusher
2025-01-18 21:21:42,131 INFO    wandb-upload_0:2577265 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/config.yaml
2025-01-18 21:21:42,147 INFO    wandb-upload_1:2577265 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/wandb-summary.json
2025-01-18 21:21:42,163 INFO    wandb-upload_4:2577265 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/output.log
2025-01-18 21:21:42,171 INFO    wandb-upload_2:2577265 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/files/requirements.txt
2025-01-18 21:21:42,223 INFO    wandb-upload_3:2577265 [upload_job.py:push():130] Uploaded file /tmp/tmplz8rujtfwandb/b03309oi-events.out.tfevents.1737252224.leonmkim-ROG-Strix-G15CS-G15CS.2577081.0
2025-01-18 21:21:42,423 INFO    Thread-11 (_thread_body):2577265 [sender.py:transition_state():614] send defer: 11
2025-01-18 21:21:42,423 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:42,423 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 11
2025-01-18 21:21:42,424 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:42,424 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 11
2025-01-18 21:21:42,424 INFO    SenderThread:2577265 [file_pusher.py:join():175] waiting for file pusher
2025-01-18 21:21:42,427 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 12
2025-01-18 21:21:42,427 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:42,427 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 12
2025-01-18 21:21:42,427 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:42,427 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 12
2025-01-18 21:21:42,427 INFO    SenderThread:2577265 [file_stream.py:finish():601] file stream finish called
2025-01-18 21:21:42,661 INFO    SenderThread:2577265 [file_stream.py:finish():605] file stream finish is done
2025-01-18 21:21:42,661 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 13
2025-01-18 21:21:42,661 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:42,661 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 13
2025-01-18 21:21:42,661 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:42,661 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 13
2025-01-18 21:21:42,661 INFO    SenderThread:2577265 [sender.py:transition_state():614] send defer: 14
2025-01-18 21:21:42,662 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: defer
2025-01-18 21:21:42,662 INFO    HandlerThread:2577265 [handler.py:handle_request_defer():184] handle defer: 14
2025-01-18 21:21:42,662 DEBUG   SenderThread:2577265 [sender.py:send():379] send: final
2025-01-18 21:21:42,662 DEBUG   SenderThread:2577265 [sender.py:send():379] send: footer
2025-01-18 21:21:42,662 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: defer
2025-01-18 21:21:42,662 INFO    SenderThread:2577265 [sender.py:send_request_defer():610] handle sender defer: 14
2025-01-18 21:21:42,662 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-18 21:21:42,663 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: poll_exit
2025-01-18 21:21:42,663 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: server_info
2025-01-18 21:21:42,663 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: server_info
2025-01-18 21:21:42,664 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: poll_exit
2025-01-18 21:21:42,665 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: get_summary
2025-01-18 21:21:42,665 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: sampled_history
2025-01-18 21:21:42,666 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: internal_messages
2025-01-18 21:21:42,719 DEBUG   SenderThread:2577265 [sender.py:send_request():406] send_request: poll_exit
2025-01-18 21:21:42,719 INFO    MainThread:2577265 [wandb_run.py:_footer_history_summary_info():4016] rendering history
2025-01-18 21:21:42,720 INFO    MainThread:2577265 [wandb_run.py:_footer_history_summary_info():4048] rendering summary
2025-01-18 21:21:42,720 INFO    MainThread:2577265 [wandb_run.py:_footer_sync_info():3975] logging synced files
2025-01-18 21:21:42,721 DEBUG   HandlerThread:2577265 [handler.py:handle_request():158] handle_request: shutdown
2025-01-18 21:21:42,721 INFO    HandlerThread:2577265 [handler.py:finish():882] shutting down handler
2025-01-18 21:21:43,665 INFO    WriterThread:2577265 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_2/2t6elatx/wandb/run-20250118_210344-2t6elatx/run-2t6elatx.wandb
2025-01-18 21:21:43,719 INFO    SenderThread:2577265 [sender.py:finish():1615] shutting down sender
2025-01-18 21:21:43,719 INFO    SenderThread:2577265 [file_pusher.py:finish():169] shutting down file pusher
2025-01-18 21:21:43,720 INFO    SenderThread:2577265 [file_pusher.py:join():175] waiting for file pusher