| 2025-01-07 18:00:40,784 INFO StreamThr :2206495 [internal.py:wandb_internal():85] W&B internal server running at pid: 2206495, started at: 2025-01-07 18:00:40.784395 |
| 2025-01-07 18:00:40,787 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 18:00:40,799 INFO WriterThread:2206495 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/run-kq00a7xd.wandb |
| 2025-01-07 18:00:40,799 DEBUG SenderThread:2206495 [sender.py:send():379] send: header |
| 2025-01-07 18:00:40,806 DEBUG SenderThread:2206495 [sender.py:send():379] send: run |
| 2025-01-07 18:00:41,054 INFO SenderThread:2206495 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files |
| 2025-01-07 18:00:41,054 INFO SenderThread:2206495 [sender.py:_start_run_threads():1188] run started: kq00a7xd with start time 1736290840.784371 |
| 2025-01-07 18:00:41,068 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 18:00:41,069 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 18:00:41,113 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 18:00:41,118 DEBUG HandlerThread:2206495 [system_info.py:__init__():26] System info init |
| 2025-01-07 18:00:41,118 DEBUG HandlerThread:2206495 [system_info.py:__init__():41] System info init done |
| 2025-01-07 18:00:41,118 INFO HandlerThread:2206495 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 18:00:41,118 INFO SystemMonitor:2206495 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 18:00:41,118 INFO HandlerThread:2206495 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 18:00:41,119 INFO SystemMonitor:2206495 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 18:00:41,120 INFO SystemMonitor:2206495 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 18:00:41,120 INFO SystemMonitor:2206495 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 18:00:41,121 INFO SystemMonitor:2206495 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 18:00:41,121 INFO SystemMonitor:2206495 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 18:00:41,131 DEBUG HandlerThread:2206495 [system_info.py:probe():152] Probing system |
| 2025-01-07 18:00:41,132 DEBUG HandlerThread:2206495 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 18:00:41,137 DEBUG HandlerThread:2206495 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 18:00:41,137 DEBUG HandlerThread:2206495 [system_info.py:probe():200] Probing system done |
| 2025-01-07 18:00:41,137 DEBUG HandlerThread:2206495 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T23:00:41.131479', 'startedAt': '2025-01-07T23:00:40.774898', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4122.291499999999, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.584, 'min': 800.0, 'max': 4700.0}, {'current': 4497.364, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 796.296257019043}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 18:00:41,137 INFO HandlerThread:2206495 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 18:00:41,137 INFO HandlerThread:2206495 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 18:00:41,137 DEBUG HandlerThread:2206495 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 18:00:41,139 DEBUG HandlerThread:2206495 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 18:00:41,139 DEBUG HandlerThread:2206495 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 18:00:41,200 DEBUG HandlerThread:2206495 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 18:00:41,201 INFO HandlerThread:2206495 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 18:00:41,204 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:00:41,204 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 18:00:41,204 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 18:00:41,204 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 18:00:41,433 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 18:00:41,433 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 18:00:41,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:41,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:00:41,435 DEBUG SenderThread:2206495 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:00:41,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:00:41,443 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/0c0rsn34-wandb-metadata.json |
| 2025-01-07 18:00:41,469 INFO wandb-upload_2:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/uxde6usb-diff.patch |
| 2025-01-07 18:00:41,477 INFO HandlerThread:2206495 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/tb" |
| save: true |
| } |
| control { |
| relay_id: "b4011873ceab42ec84f50f4786cd5445" |
| } |
| _info { |
| stream_id: "kq00a7xd" |
| } |
|
|
| 2025-01-07 18:00:41,477 DEBUG HandlerThread:2206495 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 18:00:41,524 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/23yb9kz3-code/FISH/eval_robot.py |
| 2025-01-07 18:00:41,552 DEBUG SenderThread:2206495 [sender.py:send():379] send: tbrecord |
| 2025-01-07 18:00:41,553 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:00:41,553 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 with policy live |
| 2025-01-07 18:00:41,754 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/uyy54zab-events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-metadata.json |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/diff.patch |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/code/FISH/eval_robot.py |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/requirements.txt |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:42,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/code/FISH |
| 2025-01-07 18:00:42,057 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/code |
| 2025-01-07 18:00:42,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:43,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:44,055 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:44,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:45,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:46,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:00:46,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:47,008 DEBUG SenderThread:2206495 [sender.py:send():379] send: config |
| 2025-01-07 18:00:47,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:48,056 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:48,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:49,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:50,057 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:50,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:51,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:00:51,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:52,057 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:52,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:53,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:54,058 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:54,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:55,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:56,059 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:00:56,433 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:00:56,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:00:56,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:56,552 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:00:57,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:58,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:00:59,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:00,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:01,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:02,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:02,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:03,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:04,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:05,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:06,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:07,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:07,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:08,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:09,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:10,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:11,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:01:11,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:01:11,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:12,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:12,573 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:13,063 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/config.yaml |
| 2025-01-07 18:01:13,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:14,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:15,439 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:16,439 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:17,439 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:18,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:18,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:19,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:20,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:21,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:22,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:23,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:24,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:24,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:25,440 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:26,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:01:26,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:01:26,441 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:27,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:28,067 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:28,441 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:29,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:30,068 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:30,122 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:30,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:31,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:32,069 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:32,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:33,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:34,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:35,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:35,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:36,442 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:37,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:38,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:39,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:40,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:41,121 DEBUG SystemMonitor:2206495 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 18:01:41,122 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:01:41,122 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:41,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:01:41,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:01:41,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:42,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:43,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:44,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:45,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:46,072 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:46,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:46,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:47,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:48,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:49,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:50,074 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:50,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:51,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:51,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:52,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:53,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:54,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:55,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:56,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:01:56,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:01:56,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:56,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:01:56,767 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:01:56,767 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_79f998250d76be901393.mp4 with policy now |
| 2025-01-07 18:01:56,767 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:01:56,768 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:01:56,768 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:01:56,769 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:01:57,030 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/4bd47ty3-media/videos/eval/0_eval_0_79f998250d76be901393.mp4 |
| 2025-01-07 18:01:57,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:01:57,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/0_eval_0_79f998250d76be901393.mp4 |
| 2025-01-07 18:01:57,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos |
| 2025-01-07 18:01:57,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media |
| 2025-01-07 18:01:57,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval |
| 2025-01-07 18:01:57,353 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/n69nlqr1-media/videos/eval/0_eval_0_79f998250d76be901393.mp4 |
| 2025-01-07 18:01:57,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:58,076 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:01:58,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:01:59,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:00,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:01,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:02,077 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:02,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:02,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:03,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:04,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:05,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:06,079 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:06,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:07,079 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:02:07,347 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/tjryokvc-events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:02:07,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:07,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:08,079 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:08,448 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:09,448 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:10,080 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:10,451 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:11,122 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:02:11,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:02:11,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:02:11,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:12,081 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:12,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:12,500 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:13,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:14,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:15,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:16,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:17,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:18,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:18,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:19,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:20,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:21,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:22,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:23,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:23,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:24,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:25,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:26,434 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:02:26,434 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:02:26,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:27,455 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:28,455 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:28,551 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:29,456 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:30,456 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:31,456 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:32,457 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:33,457 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:34,444 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:34,459 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:35,462 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:36,459 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:37,459 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:38,089 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:38,456 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:02:38,456 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 with policy now |
| 2025-01-07 18:02:38,459 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:02:38,460 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:02:38,460 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:02:38,460 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:02:38,461 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:38,724 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/ofn66gmw-media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 |
| 2025-01-07 18:02:39,090 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:02:39,090 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 |
| 2025-01-07 18:02:39,090 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval |
| 2025-01-07 18:02:39,378 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/6vngklpe-media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 |
| 2025-01-07 18:02:39,460 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:39,510 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:40,090 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:40,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:41,090 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:41,123 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:02:41,443 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:02:41,443 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:02:41,489 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:42,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:43,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:44,091 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:44,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:44,988 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:45,465 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:46,092 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:02:46,465 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:47,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:48,465 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:49,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:50,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:50,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:51,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:52,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:53,466 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:54,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:55,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:02:55,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:56,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:02:56,436 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:02:56,481 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:57,467 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:58,468 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:02:59,468 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:00,468 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:00,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:01,468 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:02,468 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:03,098 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:03:03,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:03,924 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:03:03,924 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:03:03,924 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:03:04,098 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:03:04,098 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:04,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:05,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:05,925 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:06,099 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:06,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:07,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:08,100 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:08,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:09,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:10,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:10,994 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:11,123 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:03:11,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:03:11,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:03:11,477 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:12,479 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:13,482 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:14,482 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:15,482 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:16,445 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:16,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:17,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:18,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:19,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:20,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:21,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:21,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:22,483 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:23,485 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:24,485 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:25,485 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:26,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:03:26,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:03:26,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:26,561 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:27,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:28,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:29,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:30,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:31,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:32,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:32,486 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:33,487 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:34,490 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:34,771 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:03:34,771 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 with policy now |
| 2025-01-07 18:03:34,774 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:03:34,774 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:03:34,774 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:03:34,775 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:03:35,044 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/1lad65yv-media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 |
| 2025-01-07 18:03:35,109 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:03:35,109 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 |
| 2025-01-07 18:03:35,109 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval |
| 2025-01-07 18:03:35,403 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/jj1jq58q-media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 |
| 2025-01-07 18:03:35,490 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:36,109 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:36,488 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:37,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:37,500 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:38,502 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:39,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:40,110 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:40,514 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:41,124 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:03:41,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:03:41,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:03:41,501 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:42,492 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:42,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:43,501 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:43,752 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:03:43,752 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:03:43,752 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:03:44,112 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:03:44,112 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:44,112 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:03:44,384 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/lqehp4vu-events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:03:44,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:45,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:46,113 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:46,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:47,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:48,114 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:48,281 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:48,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:49,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:50,114 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:03:50,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:51,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:52,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:53,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:53,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:54,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:55,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:56,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:03:56,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:03:56,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:57,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:58,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:03:58,531 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:03:59,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:00,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:01,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:02,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:03,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:04,119 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:04,257 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:04,504 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:05,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:06,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:07,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:08,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:09,447 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:09,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:10,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:11,124 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:04:11,435 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:04:11,435 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:04:11,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:12,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:13,505 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:14,513 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:14,793 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:15,089 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:04:15,089 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 with policy now |
| 2025-01-07 18:04:15,092 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:04:15,092 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:04:15,092 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:04:15,092 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:04:15,123 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:04:15,123 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 |
| 2025-01-07 18:04:15,123 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval |
| 2025-01-07 18:04:15,366 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/b3sruz0y-media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 |
| 2025-01-07 18:04:15,510 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:15,641 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/fuqlh64b-media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 |
| 2025-01-07 18:04:16,123 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:16,510 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:17,516 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:18,510 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:19,513 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:19,869 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:20,125 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:20,509 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:21,517 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:22,519 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:23,519 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:24,126 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:24,516 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:25,517 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:25,662 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:26,446 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:04:26,446 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:04:26,520 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:27,520 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:28,523 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:29,522 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:30,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:31,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:31,521 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:32,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:32,811 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:04:32,811 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:04:32,811 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:04:33,129 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:04:33,129 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:04:33,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:34,129 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:34,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:35,527 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:36,130 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:36,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:36,527 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:37,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:38,131 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:38,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:39,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:40,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:41,125 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:04:41,436 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:04:41,437 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:04:41,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:41,591 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:42,524 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:43,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:44,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:45,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:46,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:47,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:47,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:48,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:49,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:50,135 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:50,525 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:51,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:52,135 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:04:52,452 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:52,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:53,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:54,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:55,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:56,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:04:56,437 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:04:56,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:57,498 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:04:57,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:58,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:04:59,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:00,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:01,526 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:02,527 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:03,453 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:03,527 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:04,539 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:04,970 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:05:04,971 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 with policy now |
| 2025-01-07 18:05:04,973 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:05:04,974 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:05:04,974 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:04,974 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:05,140 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:05,140 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 |
| 2025-01-07 18:05:05,140 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval |
| 2025-01-07 18:05:05,247 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/dfrn8fuh-media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 |
| 2025-01-07 18:05:05,529 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/etj2slca-media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 |
| 2025-01-07 18:05:05,538 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:06,140 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:06,534 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:07,536 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:08,469 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:08,543 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:09,536 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:10,142 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:10,537 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:11,125 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:05:11,438 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:05:11,439 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:05:11,540 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:12,541 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:13,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:13,601 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:14,143 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:14,556 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:15,552 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:16,559 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:17,548 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:18,548 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:19,133 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:19,145 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:19,428 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/9wvzz7mh-events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:19,544 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:19,868 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:05:19,869 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:19,869 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,090 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:05:20,090 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:05:20,145 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:20,145 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:20,145 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:20,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:05:20,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media |
| 2025-01-07 18:05:20,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval |
| 2025-01-07 18:05:20,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table |
| 2025-01-07 18:05:20,232 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:05:20,342 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/f853rohi-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:05:20,548 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:20,569 INFO wandb-upload_1:2206495 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmplfox_w0_ |
| 2025-01-07 18:05:20,654 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:05:20,662 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:05:20,942 INFO SenderThread:2206495 [sender.py:send_request_log_artifact():1525] logged artifact run-kq00a7xd-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjA0ODY3Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTczNzE5OQ==', 'latestArtifact': None}} |
| 2025-01-07 18:05:20,942 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:05:20,942 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-07 18:05:20,942 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:05:20,942 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,943 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,943 DEBUG SenderThread:2206495 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:05:20,943 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,947 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,947 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,947 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,948 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,948 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,948 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,948 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,949 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,949 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,949 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,949 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,950 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,950 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,950 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,950 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,951 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,951 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,951 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,951 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,952 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:20,952 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:20,952 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:05:21,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:21,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:21,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json |
| 2025-01-07 18:05:21,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval |
| 2025-01-07 18:05:21,211 INFO wandb-upload_2:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/1gg6gwtw-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:05:21,323 INFO wandb-upload_1:2206495 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp6ju784l7 |
| 2025-01-07 18:05:21,357 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/k5py8w0s-media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json |
| 2025-01-07 18:05:21,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:21,725 INFO SenderThread:2206495 [sender.py:send_request_log_artifact():1525] logged artifact run-kq00a7xd-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjA0ODc3MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTczNzI3OA==', 'latestArtifact': None}} |
| 2025-01-07 18:05:21,726 DEBUG SenderThread:2206495 [sender.py:send():379] send: files |
| 2025-01-07 18:05:21,726 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json with policy now |
| 2025-01-07 18:05:21,726 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:05:21,726 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:21,727 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:21,978 INFO wandb-upload_2:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/9gsncjd5-media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json |
| 2025-01-07 18:05:22,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:22,146 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:22,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:23,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:24,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:24,728 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:25,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:26,437 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:05:26,437 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:05:26,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:27,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:28,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:29,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:30,503 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:30,545 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:31,150 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/config.yaml |
| 2025-01-07 18:05:31,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:32,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:33,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:34,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:35,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:36,457 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:36,546 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:37,547 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:38,581 DEBUG SenderThread:2206495 [sender.py:send():379] send: exit |
| 2025-01-07 18:05:38,582 INFO SenderThread:2206495 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 18:05:38,582 INFO SenderThread:2206495 [sender.py:send_exit():588] handling runtime: 297 |
| 2025-01-07 18:05:38,582 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:38,583 INFO SenderThread:2206495 [sender.py:send_exit():594] send defer |
| 2025-01-07 18:05:38,583 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:38,583 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 18:05:38,583 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:38,583 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 18:05:38,583 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 18:05:38,583 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:38,583 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 18:05:38,583 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:38,583 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 18:05:38,584 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 18:05:38,584 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:38,584 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 18:05:38,584 INFO HandlerThread:2206495 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 18:05:38,584 DEBUG SystemMonitor:2206495 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 18:05:38,584 DEBUG SystemMonitor:2206495 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 18:05:38,584 INFO HandlerThread:2206495 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 18:05:38,585 INFO HandlerThread:2206495 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 18:05:38,654 INFO HandlerThread:2206495 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 18:05:38,655 INFO HandlerThread:2206495 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 18:05:38,655 INFO HandlerThread:2206495 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 18:05:38,655 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:38,655 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 18:05:38,655 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 18:05:38,655 DEBUG SenderThread:2206495 [sender.py:send():379] send: stats |
| 2025-01-07 18:05:38,656 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:38,656 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 18:05:38,656 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:38,656 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 18:05:38,656 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 18:05:38,656 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:38,656 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 18:05:39,153 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:40,153 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:44,876 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:05:44,876 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:44,876 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 18:05:44,877 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:44,877 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 18:05:44,878 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 18:05:44,878 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:05:44,878 DEBUG SenderThread:2206495 [sender.py:send():379] send: history |
| 2025-01-07 18:05:44,878 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:44,878 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 18:05:44,878 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:05:44,879 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:44,879 DEBUG SenderThread:2206495 [sender.py:send():379] send: summary |
| 2025-01-07 18:05:44,879 INFO SenderThread:2206495 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:05:44,880 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:44,880 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 18:05:44,880 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 18:05:44,880 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:44,880 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 18:05:44,880 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:44,880 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 18:05:44,880 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 18:05:44,880 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:05:44,880 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:44,880 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 18:05:44,880 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:44,880 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 18:05:45,155 INFO Thread-12 :2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:45,583 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:05:46,623 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 18:05:46,623 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:05:46,624 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:46,624 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 18:05:46,624 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:46,624 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 18:05:46,624 INFO SenderThread:2206495 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 18:05:46,626 INFO SenderThread:2206495 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 18:05:46,628 INFO SenderThread:2206495 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 18:05:46,642 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 18:05:46,642 DEBUG SenderThread:2206495 [sender.py:send():379] send: artifact |
| 2025-01-07 18:05:46,642 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:46,645 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 18:05:46,829 INFO SenderThread:2206495 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTg2NDkwOA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'versionIndex': 436}}} |
| 2025-01-07 18:05:46,829 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:46,829 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 18:05:46,829 INFO SenderThread:2206495 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/config.yaml config.yaml |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/requirements.txt requirements.txt |
| 2025-01-07 18:05:47,157 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 18:05:47,158 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log output.log |
| 2025-01-07 18:05:47,158 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/diff.patch diff.patch |
| 2025-01-07 18:05:47,158 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 18:05:47,160 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 media/videos/eval/3_eval_3_796eb7d92e1e7dcc5b92.mp4 |
| 2025-01-07 18:05:47,160 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 media/videos/eval/2_eval_2_a16b4848a19dfb390151.mp4 |
| 2025-01-07 18:05:47,160 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/0_eval_0_79f998250d76be901393.mp4 media/videos/eval/0_eval_0_79f998250d76be901393.mp4 |
| 2025-01-07 18:05:47,160 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 media/videos/eval/1_eval_1_5777bf060ba6d7d77b45.mp4 |
| 2025-01-07 18:05:47,160 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 media/videos/eval/4_eval_4_227b80e6fd7f10383222.mp4 |
| 2025-01-07 18:05:47,166 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json media/table/eval/total_success_rate_ci_6_2f3f136f2937a5b3202d.table.json |
| 2025-01-07 18:05:47,166 INFO SenderThread:2206495 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:05:47,166 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 18:05:47,167 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:47,167 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 18:05:47,167 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:47,167 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 18:05:47,167 INFO SenderThread:2206495 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:05:47,339 INFO wandb-upload_0:2206495 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/config.yaml |
| 2025-01-07 18:05:47,391 INFO wandb-upload_2:2206495 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/output.log |
| 2025-01-07 18:05:47,391 INFO wandb-upload_1:2206495 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/wandb-summary.json |
| 2025-01-07 18:05:47,392 INFO wandb-upload_4:2206495 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/files/requirements.txt |
| 2025-01-07 18:05:47,434 INFO wandb-upload_3:2206495 [upload_job.py:push():130] Uploaded file /tmp/tmpl7qgw5g_wandb/375vrvdc-events.out.tfevents.1736290841.leonmkim-ROG-Strix-G15CS-G15CS.2206328.0 |
| 2025-01-07 18:05:47,584 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:05:47,584 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:05:47,634 INFO Thread-11 (_thread_body):2206495 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 18:05:47,635 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:47,635 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 18:05:47,635 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:47,635 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 18:05:47,635 INFO SenderThread:2206495 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 18:05:47,636 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 18:05:47,637 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:47,637 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 18:05:47,637 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:47,637 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 18:05:47,637 INFO SenderThread:2206495 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 18:05:47,849 INFO SenderThread:2206495 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 18:05:47,849 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 18:05:47,849 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:47,849 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 18:05:47,849 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:47,849 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 18:05:47,849 INFO SenderThread:2206495 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 18:05:47,849 DEBUG SenderThread:2206495 [sender.py:send():379] send: final |
| 2025-01-07 18:05:47,849 DEBUG SenderThread:2206495 [sender.py:send():379] send: footer |
| 2025-01-07 18:05:47,849 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:05:47,850 INFO HandlerThread:2206495 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 18:05:47,850 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:05:47,850 INFO SenderThread:2206495 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 18:05:47,850 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:05:47,850 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 18:05:47,850 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:05:47,851 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:05:47,851 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 18:05:47,852 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 18:05:47,853 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 18:05:47,854 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:05:47,904 DEBUG SenderThread:2206495 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:05:47,904 INFO MainThread:2206495 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 18:05:47,905 INFO MainThread:2206495 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 18:05:47,906 INFO MainThread:2206495 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 18:05:47,906 DEBUG HandlerThread:2206495 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 18:05:47,906 INFO HandlerThread:2206495 [handler.py:finish():882] shutting down handler |
| 2025-01-07 18:05:48,851 INFO WriterThread:2206495 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/kq00a7xd/wandb/run-20250107_180040-kq00a7xd/run-kq00a7xd.wandb |
| 2025-01-07 18:05:48,904 INFO SenderThread:2206495 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 18:05:48,904 INFO SenderThread:2206495 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:05:48,904 INFO SenderThread:2206495 [file_pusher.py:join():175] waiting for file pusher |
|
|
