| 2024-12-11 19:13:40,902 INFO StreamThr :773446 [internal.py:wandb_internal():85] W&B internal server running at pid: 773446, started at: 2024-12-11 19:13:40.902318 |
| 2024-12-11 19:13:40,903 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status |
| 2024-12-11 19:13:40,918 INFO WriterThread:773446 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/run-l07wa4hd.wandb |
| 2024-12-11 19:13:40,918 DEBUG SenderThread:773446 [sender.py:send():379] send: header |
| 2024-12-11 19:13:40,926 DEBUG SenderThread:773446 [sender.py:send():379] send: run |
| 2024-12-11 19:13:41,175 INFO SenderThread:773446 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files |
| 2024-12-11 19:13:41,175 INFO SenderThread:773446 [sender.py:_start_run_threads():1188] run started: l07wa4hd with start time 1733962420.904478 |
| 2024-12-11 19:13:41,191 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-11 19:13:41,192 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: check_version |
| 2024-12-11 19:13:41,268 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-11 19:13:41,273 DEBUG HandlerThread:773446 [system_info.py:__init__():26] System info init |
| 2024-12-11 19:13:41,273 DEBUG HandlerThread:773446 [system_info.py:__init__():41] System info init done |
| 2024-12-11 19:13:41,273 INFO HandlerThread:773446 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-11 19:13:41,273 INFO SystemMonitor:773446 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-11 19:13:41,273 INFO HandlerThread:773446 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-11 19:13:41,273 INFO SystemMonitor:773446 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-11 19:13:41,274 INFO SystemMonitor:773446 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-11 19:13:41,275 INFO SystemMonitor:773446 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-11 19:13:41,275 INFO SystemMonitor:773446 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-11 19:13:41,276 INFO SystemMonitor:773446 [interfaces.py:start():188] Started network monitoring |
| 2024-12-11 19:13:41,286 DEBUG HandlerThread:773446 [system_info.py:probe():152] Probing system |
| 2024-12-11 19:13:41,288 DEBUG HandlerThread:773446 [system_info.py:_probe_git():137] Probing git |
| 2024-12-11 19:13:41,293 DEBUG HandlerThread:773446 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-11 19:13:41,293 DEBUG HandlerThread:773446 [system_info.py:probe():200] Probing system done |
| 2024-12-11 19:13:41,293 DEBUG HandlerThread:773446 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-12T00:13:41.286723', 'startedAt': '2024-12-12T00:13:40.893574', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '1b557f164c7f62a8d1545c761b13ede1677c3a83'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.654375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.045, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 705.1167831420898}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71596908569336}} |
| 2024-12-11 19:13:41,293 INFO HandlerThread:773446 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-11 19:13:41,293 INFO HandlerThread:773446 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-11 19:13:41,293 DEBUG HandlerThread:773446 [system_info.py:_save_code():44] Saving code |
| 2024-12-11 19:13:41,294 DEBUG HandlerThread:773446 [system_info.py:_save_code():67] Saving code done |
| 2024-12-11 19:13:41,294 DEBUG HandlerThread:773446 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-11 19:13:41,362 DEBUG HandlerThread:773446 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-11 19:13:41,363 INFO HandlerThread:773446 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-11 19:13:41,366 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:13:41,366 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-11 19:13:41,366 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-11 19:13:41,367 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-11 19:13:41,616 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-11 19:13:41,616 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-11 19:13:41,616 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:13:41,617 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:13:41,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:41,619 INFO HandlerThread:773446 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/tb" |
| save: true |
| } |
| control { |
| relay_id: "85246c7cda7d49f9afbefde503c903d6" |
| } |
| _info { |
| stream_id: "l07wa4hd" |
| } |
|
|
| 2024-12-11 19:13:41,619 DEBUG HandlerThread:773446 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-11 19:13:41,639 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/r8wlspo2-wandb-metadata.json |
| 2024-12-11 19:13:41,640 INFO wandb-upload_2:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/azg7zrgd-diff.patch |
| 2024-12-11 19:13:41,683 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/3fipiwh0-code/FISH/eval_robot.py |
| 2024-12-11 19:13:41,705 DEBUG SenderThread:773446 [sender.py:send():379] send: telemetry |
| 2024-12-11 19:13:41,705 DEBUG SenderThread:773446 [sender.py:send():379] send: tbrecord |
| 2024-12-11 19:13:41,705 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:13:41,705 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 with policy live |
| 2024-12-11 19:13:41,892 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/77gm1cue-events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:13:42,177 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:13:42,177 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/requirements.txt |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/code/FISH/eval_robot.py |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-metadata.json |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/diff.patch |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/code |
| 2024-12-11 19:13:42,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/code/FISH |
| 2024-12-11 19:13:42,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:43,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:44,176 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:13:44,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:45,501 DEBUG SenderThread:773446 [sender.py:send():379] send: config |
| 2024-12-11 19:13:45,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:46,507 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:13:46,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:47,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:48,178 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:13:48,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:49,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:50,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:51,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:13:51,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:52,179 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:13:52,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:53,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:54,179 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:13:54,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:55,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:56,616 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:13:56,616 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:13:56,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:56,722 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:13:57,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:58,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:13:59,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:00,181 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:00,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:01,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:02,182 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:02,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:02,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:03,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:04,182 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:04,619 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:05,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:06,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:07,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:08,011 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:08,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:09,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:10,184 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:10,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:11,616 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:14:11,617 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:14:11,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:12,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:13,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:13,745 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:14,185 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/config.yaml |
| 2024-12-11 19:14:14,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:15,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:16,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:17,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:18,186 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:18,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:19,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:19,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:20,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:21,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:22,187 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:22,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:23,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:24,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:24,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:25,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:26,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:14:26,617 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:14:26,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:27,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:28,623 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:29,326 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:14:29,327 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 with policy now |
| 2024-12-11 19:14:29,327 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 19:14:29,328 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:14:29,328 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:14:29,329 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:14:29,595 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/p9oiypsh-media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 |
| 2024-12-11 19:14:29,623 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos |
| 2024-12-11 19:14:30,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media |
| 2024-12-11 19:14:30,332 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:30,478 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/otzzej9i-media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 |
| 2024-12-11 19:14:30,625 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:31,626 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:32,190 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:32,627 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:33,627 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:34,191 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:34,627 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:35,627 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:35,886 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:36,192 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:36,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:37,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:38,192 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:14:38,192 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:38,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:39,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:40,193 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:40,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:40,939 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:41,279 DEBUG SystemMonitor:773446 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-11 19:14:41,280 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:14:41,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:14:41,617 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:14:41,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:42,194 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:42,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:43,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:44,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:45,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:46,558 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:46,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:47,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:48,196 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:48,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:49,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:50,196 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:50,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:51,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:51,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:52,197 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:52,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:53,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:54,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:55,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:56,617 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:14:56,617 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:14:56,658 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:56,709 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:14:57,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:58,199 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:14:58,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:14:59,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:00,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:01,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:02,618 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:02,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:03,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:04,632 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:05,632 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:06,632 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:07,633 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:08,326 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:08,633 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:09,596 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:15:09,596 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 with policy now |
| 2024-12-11 19:15:09,597 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 19:15:09,598 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:15:09,598 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:15:09,600 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:15:09,636 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:09,849 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/9qh203tq-media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 |
| 2024-12-11 19:15:10,203 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:15:10,203 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:10,203 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 |
| 2024-12-11 19:15:10,203 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval |
| 2024-12-11 19:15:10,471 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/4acat9m8-media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 |
| 2024-12-11 19:15:10,638 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:11,280 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:15:11,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:15:11,622 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:15:11,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:12,204 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:12,637 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:13,550 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:13,639 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:14,204 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:14,640 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:15,643 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:16,205 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:16,647 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:17,647 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:18,205 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:18,626 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:18,650 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:19,647 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:20,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:21,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:22,652 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:23,647 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:24,624 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:24,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:25,659 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:26,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:15:26,620 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:15:26,661 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:27,652 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:28,650 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:29,651 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:29,733 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:30,655 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:31,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:32,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:33,652 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:34,655 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:35,634 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:35,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:36,653 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:37,658 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:38,657 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:39,659 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:40,634 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:40,656 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:41,281 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:15:41,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:15:41,620 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:15:41,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:42,659 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:43,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:44,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:45,664 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:45,739 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:46,215 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:15:46,451 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/7wdb8k1w-events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:15:46,670 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:47,040 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:15:47,040 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:15:47,040 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:15:47,215 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:15:47,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:48,216 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:48,669 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:49,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:50,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:50,978 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:51,685 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:52,217 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:52,670 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:53,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:54,218 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:15:54,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:55,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:56,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:15:56,620 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:15:56,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:56,737 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:15:57,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:58,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:15:59,665 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:00,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:01,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:02,087 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:02,220 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:02,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:03,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:04,221 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:04,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:05,666 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:06,222 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:06,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:07,628 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:07,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:08,222 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:08,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:09,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:10,223 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:10,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:11,281 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:16:11,620 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:16:11,621 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:16:11,667 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:12,224 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:12,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:12,691 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:13,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:14,224 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:14,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:15,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:16,225 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:16,669 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:17,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:18,226 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:18,286 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:18,668 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:19,669 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:20,227 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:20,673 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:20,921 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:16:20,921 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 with policy now |
| 2024-12-11 19:16:20,922 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 19:16:20,922 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:16:20,922 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:16:20,923 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:16:21,212 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/wobof1o8-media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 |
| 2024-12-11 19:16:21,227 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:16:21,227 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 |
| 2024-12-11 19:16:21,227 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval |
| 2024-12-11 19:16:21,502 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/a144xsb1-media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 |
| 2024-12-11 19:16:21,676 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:22,227 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:22,680 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:23,633 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:23,693 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:24,675 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:25,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:26,229 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:26,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:16:26,621 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:16:26,672 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:27,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:28,229 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:28,681 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:28,689 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:29,688 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:30,689 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:31,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:32,682 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:33,685 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:33,879 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:16:33,882 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:16:33,882 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:33,883 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:16:34,231 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:16:34,231 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:16:34,232 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:34,683 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:35,685 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:36,232 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:36,681 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:37,684 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:38,233 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:38,683 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:39,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:39,778 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:40,233 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:40,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:41,282 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:16:41,622 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:16:41,622 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:16:41,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:42,234 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:42,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:43,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:44,235 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:44,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:45,438 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:45,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:46,235 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:46,677 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:47,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:48,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:49,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:50,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:50,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:51,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:52,238 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:52,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:53,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:54,238 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:54,678 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:55,638 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:16:55,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:56,239 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:56,621 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:16:56,621 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:16:56,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:57,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:58,240 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:16:58,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:16:59,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:00,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:01,442 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:01,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:02,241 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:02,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:03,679 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:04,242 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:04,680 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:05,680 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:06,244 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:06,630 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:06,680 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:07,690 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:08,079 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:17:08,079 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 with policy now |
| 2024-12-11 19:17:08,082 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 19:17:08,082 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:17:08,082 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:17:08,083 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:17:08,244 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:17:08,244 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:08,245 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 |
| 2024-12-11 19:17:08,245 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval |
| 2024-12-11 19:17:08,353 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/q8qglt22-media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 |
| 2024-12-11 19:17:08,660 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/ac716fp2-media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 |
| 2024-12-11 19:17:08,691 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:09,694 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:10,245 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:10,696 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:11,282 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:17:11,629 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:17:11,629 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:17:11,688 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:11,715 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:12,246 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:12,699 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:13,692 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:14,695 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:15,697 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:16,247 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:17:16,247 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:16,525 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/6mzwuq1v-events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:17:16,699 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:16,933 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:17:16,933 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:17:16,933 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:16,937 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:17:17,247 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:17:17,696 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:18,248 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:18,700 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:19,702 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:20,702 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:21,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:22,173 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:22,249 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:22,698 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:23,702 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:24,250 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:24,703 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:25,702 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:26,251 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:26,631 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:17:26,631 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:17:26,704 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:27,703 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:27,728 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:28,251 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:28,704 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:29,704 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:30,749 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:31,706 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:32,253 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:32,708 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:33,637 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:33,709 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:34,708 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:35,709 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:36,713 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:37,710 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:38,256 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:38,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:39,637 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:39,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:40,712 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:41,283 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:17:41,627 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:17:41,627 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:17:41,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:42,257 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:42,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:43,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:44,710 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:44,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:45,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:46,711 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:47,712 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:48,712 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:49,712 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:50,146 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:50,260 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:50,712 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:51,645 DEBUG SenderThread:773446 [sender.py:send():379] send: files |
| 2024-12-11 19:17:51,645 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 with policy now |
| 2024-12-11 19:17:51,646 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 19:17:51,646 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:17:51,646 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:17:51,648 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:17:51,722 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:52,260 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:17:52,260 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:52,261 INFO Thread-12 :773446 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 |
| 2024-12-11 19:17:52,261 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval |
| 2024-12-11 19:17:52,519 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/lxp6kvg9-media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 |
| 2024-12-11 19:17:52,722 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:52,817 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/dmphh3x2-media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 |
| 2024-12-11 19:17:53,726 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:54,261 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:54,727 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:55,538 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:17:55,733 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:56,262 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:56,634 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:17:56,634 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:17:56,734 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:57,750 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:58,262 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:17:58,757 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:59,752 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:17:59,983 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:17:59,983 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:17:59,984 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:18:00,263 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:18:00,263 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:18:00,747 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:00,985 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:01,753 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:02,264 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:18:02,747 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:03,745 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:04,752 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:05,751 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:06,660 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:06,749 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:07,750 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:08,747 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:09,760 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:10,755 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:11,283 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:18:11,639 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 19:18:11,640 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 19:18:11,753 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:11,754 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:12,754 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:13,758 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:14,755 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:15,762 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:16,006 DEBUG SenderThread:773446 [sender.py:send():379] send: exit |
| 2024-12-11 19:18:16,006 INFO SenderThread:773446 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-11 19:18:16,006 INFO SenderThread:773446 [sender.py:send_exit():588] handling runtime: 274 |
| 2024-12-11 19:18:16,006 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:18:16,006 INFO SenderThread:773446 [sender.py:send_exit():594] send defer |
| 2024-12-11 19:18:16,006 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:16,006 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-11 19:18:16,007 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:16,007 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-11 19:18:16,007 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-11 19:18:16,007 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:16,007 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-11 19:18:16,007 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:16,007 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-11 19:18:16,007 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-11 19:18:16,007 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:16,007 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-11 19:18:16,007 INFO HandlerThread:773446 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-11 19:18:16,007 DEBUG SystemMonitor:773446 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-11 19:18:16,007 INFO HandlerThread:773446 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-11 19:18:16,007 DEBUG SystemMonitor:773446 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-11 19:18:16,008 INFO HandlerThread:773446 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-11 19:18:16,063 INFO HandlerThread:773446 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-11 19:18:16,064 INFO HandlerThread:773446 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-11 19:18:16,064 INFO HandlerThread:773446 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-11 19:18:16,064 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:16,064 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-11 19:18:16,064 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-11 19:18:16,064 DEBUG SenderThread:773446 [sender.py:send():379] send: stats |
| 2024-12-11 19:18:16,064 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:16,064 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-11 19:18:16,065 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:16,065 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-11 19:18:16,065 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-11 19:18:16,065 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:16,065 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-11 19:18:16,268 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:18:18,269 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:18:22,985 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:22,985 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:22,985 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-11 19:18:22,985 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:22,985 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:22,986 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-11 19:18:22,986 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-11 19:18:22,986 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:22,986 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:22,986 DEBUG SenderThread:773446 [sender.py:send():379] send: history |
| 2024-12-11 19:18:22,986 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-11 19:18:22,986 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:18:22,988 DEBUG SenderThread:773446 [sender.py:send():379] send: summary |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 19:18:22,988 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-11 19:18:22,988 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:22,988 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-11 19:18:22,988 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-11 19:18:22,988 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-11 19:18:22,989 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 19:18:22,989 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:22,989 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-11 19:18:22,989 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:22,989 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-11 19:18:23,007 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:23,271 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:18:24,003 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-11 19:18:24,003 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:24,003 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:24,004 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-11 19:18:24,004 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:24,004 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-11 19:18:24,004 INFO SenderThread:773446 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-11 19:18:24,006 INFO SenderThread:773446 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-11 19:18:24,007 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:24,031 INFO SenderThread:773446 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-11 19:18:24,042 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-11 19:18:24,042 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:24,042 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:24,043 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-11 19:18:24,043 DEBUG SenderThread:773446 [sender.py:send():379] send: artifact |
| 2024-12-11 19:18:24,271 INFO Thread-12 :773446 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:18:24,605 INFO wandb-upload_2:773446 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpclrh1q5h |
| 2024-12-11 19:18:24,605 INFO wandb-upload_0:773446 [upload_job.py:push():85] Skipped uploading /tmp/tmpnbwfrddq/wandb-job.json |
| 2024-12-11 19:18:24,799 INFO wandb-upload_1:773446 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdrsc03bb |
| 2024-12-11 19:18:25,008 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:25,398 INFO SenderThread:773446 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM2Nzk0NTgyMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM2Nzk0MTU0Mw==', 'versionIndex': 370}}} |
| 2024-12-11 19:18:25,398 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:25,398 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-11 19:18:25,398 INFO SenderThread:773446 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-11 19:18:26,272 INFO SenderThread:773446 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files |
| 2024-12-11 19:18:26,272 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/config.yaml config.yaml |
| 2024-12-11 19:18:26,272 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json wandb-summary.json |
| 2024-12-11 19:18:26,272 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/requirements.txt requirements.txt |
| 2024-12-11 19:18:26,274 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-11 19:18:26,274 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:18:26,279 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log output.log |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/diff.patch diff.patch |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 media/videos/eval/3_eval_3_334d1f02994a0fcdeda5.mp4 |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 media/videos/eval/4_eval_4_35283caf4103c3829cd1.mp4 |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 media/videos/eval/2_eval_2_c9cccf7ec7cae45b2fa1.mp4 |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 media/videos/eval/1_eval_1_3a45ace652f05385ee75.mp4 |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 media/videos/eval/0_eval_0_3da136561a2ad42749fd.mp4 |
| 2024-12-11 19:18:26,280 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-11 19:18:26,281 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:26,281 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:26,281 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-11 19:18:26,284 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:26,284 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-11 19:18:26,284 INFO SenderThread:773446 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-11 19:18:26,455 INFO wandb-upload_1:773446 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/wandb-summary.json |
| 2024-12-11 19:18:26,481 INFO wandb-upload_0:773446 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/requirements.txt |
| 2024-12-11 19:18:26,523 INFO wandb-upload_4:773446 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/output.log |
| 2024-12-11 19:18:26,524 INFO wandb-upload_2:773446 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/files/config.yaml |
| 2024-12-11 19:18:26,535 INFO wandb-upload_3:773446 [upload_job.py:push():130] Uploaded file /tmp/tmpaa23qkokwandb/e8bkegnr-events.out.tfevents.1733962421.leonmkim-ROG-Strix-G15CS-G15CS.773302.0 |
| 2024-12-11 19:18:26,735 INFO Thread-11 (_thread_body):773446 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-11 19:18:26,735 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:26,736 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-11 19:18:26,736 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:26,736 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-11 19:18:26,736 INFO SenderThread:773446 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-11 19:18:26,737 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-11 19:18:26,737 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:26,737 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-11 19:18:26,737 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:26,737 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-11 19:18:26,737 INFO SenderThread:773446 [file_stream.py:finish():601] file stream finish called |
| 2024-12-11 19:18:26,972 INFO SenderThread:773446 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-11 19:18:26,972 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-11 19:18:26,973 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:26,973 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-11 19:18:26,973 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:26,973 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-11 19:18:26,973 INFO SenderThread:773446 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-11 19:18:26,973 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 19:18:26,973 INFO HandlerThread:773446 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-11 19:18:26,973 DEBUG SenderThread:773446 [sender.py:send():379] send: final |
| 2024-12-11 19:18:26,973 DEBUG SenderThread:773446 [sender.py:send():379] send: footer |
| 2024-12-11 19:18:26,973 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 19:18:26,974 INFO SenderThread:773446 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-11 19:18:26,974 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:26,974 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:26,974 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-11 19:18:26,975 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: server_info |
| 2024-12-11 19:18:26,976 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 19:18:26,976 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-11 19:18:26,977 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-11 19:18:26,977 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 19:18:27,035 DEBUG SenderThread:773446 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 19:18:27,036 INFO MainThread:773446 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-11 19:18:27,036 INFO MainThread:773446 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-11 19:18:27,036 INFO MainThread:773446 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-11 19:18:27,036 DEBUG HandlerThread:773446 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-11 19:18:27,036 INFO HandlerThread:773446 [handler.py:finish():882] shutting down handler |
| 2024-12-11 19:18:27,976 INFO WriterThread:773446 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/266406_0/191334/wandb/run-20241211_191340-l07wa4hd/run-l07wa4hd.wandb |
| 2024-12-11 19:18:28,036 INFO SenderThread:773446 [sender.py:finish():1615] shutting down sender |
| 2024-12-11 19:18:28,036 INFO SenderThread:773446 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-11 19:18:28,036 INFO SenderThread:773446 [file_pusher.py:join():175] waiting for file pusher |
|
|
