| 2025-01-07 16:51:47,443 INFO StreamThr :2190370 [internal.py:wandb_internal():85] W&B internal server running at pid: 2190370, started at: 2025-01-07 16:51:47.442833 |
| 2025-01-07 16:51:47,444 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 16:51:47,460 INFO WriterThread:2190370 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/run-nuw4e53j.wandb |
| 2025-01-07 16:51:47,460 DEBUG SenderThread:2190370 [sender.py:send():379] send: header |
| 2025-01-07 16:51:47,468 DEBUG SenderThread:2190370 [sender.py:send():379] send: run |
| 2025-01-07 16:51:47,723 INFO SenderThread:2190370 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files |
| 2025-01-07 16:51:47,723 INFO SenderThread:2190370 [sender.py:_start_run_threads():1188] run started: nuw4e53j with start time 1736286707.443296 |
| 2025-01-07 16:51:47,738 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 16:51:47,738 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 16:51:47,808 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 16:51:47,813 DEBUG HandlerThread:2190370 [system_info.py:__init__():26] System info init |
| 2025-01-07 16:51:47,813 DEBUG HandlerThread:2190370 [system_info.py:__init__():41] System info init done |
| 2025-01-07 16:51:47,813 INFO HandlerThread:2190370 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 16:51:47,813 INFO SystemMonitor:2190370 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 16:51:47,813 INFO HandlerThread:2190370 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 16:51:47,814 INFO SystemMonitor:2190370 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 16:51:47,815 INFO SystemMonitor:2190370 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 16:51:47,815 INFO SystemMonitor:2190370 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 16:51:47,816 INFO SystemMonitor:2190370 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 16:51:47,816 INFO SystemMonitor:2190370 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 16:51:47,827 DEBUG HandlerThread:2190370 [system_info.py:probe():152] Probing system |
| 2025-01-07 16:51:47,829 DEBUG HandlerThread:2190370 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 16:51:47,833 DEBUG HandlerThread:2190370 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 16:51:47,834 DEBUG HandlerThread:2190370 [system_info.py:probe():200] Probing system done |
| 2025-01-07 16:51:47,834 DEBUG HandlerThread:2190370 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T21:51:47.827932', 'startedAt': '2025-01-07T21:51:47.434771', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.6253749999996, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4498.284, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.407, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 711.6432304382324}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 16:51:47,834 INFO HandlerThread:2190370 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 16:51:47,834 INFO HandlerThread:2190370 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 16:51:47,834 DEBUG HandlerThread:2190370 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 16:51:47,839 DEBUG HandlerThread:2190370 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 16:51:47,840 DEBUG HandlerThread:2190370 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 16:51:47,902 DEBUG HandlerThread:2190370 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 16:51:47,902 INFO HandlerThread:2190370 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 16:51:47,905 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:51:47,905 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 16:51:47,905 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 16:51:47,906 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 16:51:48,144 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/gni57eyu-wandb-metadata.json |
| 2025-01-07 16:51:48,161 INFO wandb-upload_2:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/f5o9qzsv-diff.patch |
| 2025-01-07 16:51:48,162 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 16:51:48,162 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 16:51:48,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:48,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:51:48,165 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:51:48,182 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/vflu5nhl-code/FISH/eval_robot.py |
| 2025-01-07 16:51:48,209 INFO HandlerThread:2190370 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/tb" |
| save: true |
| } |
| control { |
| relay_id: "f0fa338b54b44c869d5fa37855062368" |
| } |
| _info { |
| stream_id: "nuw4e53j" |
| } |
|
|
| 2025-01-07 16:51:48,209 DEBUG HandlerThread:2190370 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 16:51:48,264 DEBUG SenderThread:2190370 [sender.py:send():379] send: telemetry |
| 2025-01-07 16:51:48,276 DEBUG SenderThread:2190370 [sender.py:send():379] send: tbrecord |
| 2025-01-07 16:51:48,278 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:51:48,278 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 with policy live |
| 2025-01-07 16:51:48,453 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/wxui3ns1-events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:51:48,724 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:51:48,724 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/code/FISH/eval_robot.py |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-metadata.json |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/diff.patch |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/requirements.txt |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/code |
| 2025-01-07 16:51:48,725 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/code/FISH |
| 2025-01-07 16:51:49,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:50,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:50,724 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:51:51,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:52,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:53,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:51:53,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:54,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:55,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:56,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:57,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:58,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:51:58,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:51:59,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:00,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:01,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:02,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:03,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:52:03,163 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:52:03,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:03,264 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:04,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:05,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:06,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:07,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:08,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:08,502 DEBUG SenderThread:2190370 [sender.py:send():379] send: config |
| 2025-01-07 16:52:08,503 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:09,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:10,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:10,729 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:52:11,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:12,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:13,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:14,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:14,281 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:14,730 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:52:15,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:16,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:17,167 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:18,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:52:18,163 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:52:18,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:18,731 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:52:19,167 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:20,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:20,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:20,732 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/config.yaml |
| 2025-01-07 16:52:21,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:22,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:23,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:24,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:25,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:25,654 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:26,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:26,733 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:52:27,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:28,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:29,169 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:30,169 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:30,691 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:30,735 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:52:31,169 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:32,169 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:33,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:52:33,163 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:52:33,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:34,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:35,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:36,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:36,228 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:37,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:38,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:39,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:40,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:41,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:42,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:42,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:43,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:44,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:45,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:46,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:47,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:47,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:47,817 DEBUG SystemMonitor:2190370 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 16:52:47,817 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:52:48,163 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:52:48,164 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:52:48,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:49,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:50,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:51,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:52,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:52,285 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:53,172 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:54,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:55,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:56,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:57,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:58,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:52:58,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:52:59,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:00,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:01,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:02,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:03,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:53:03,164 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:53:03,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:03,220 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:04,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:04,281 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:53:04,282 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:53:04,282 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 with policy now |
| 2025-01-07 16:53:04,282 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:53:04,282 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:53:04,283 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:53:04,559 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/ze03nlca-media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 |
| 2025-01-07 16:53:04,744 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:04,744 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 |
| 2025-01-07 16:53:04,744 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:53:04,744 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval |
| 2025-01-07 16:53:04,744 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos |
| 2025-01-07 16:53:04,745 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media |
| 2025-01-07 16:53:05,009 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/lgwa2lah-media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 |
| 2025-01-07 16:53:05,176 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:06,178 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:06,745 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:07,178 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:08,178 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:08,745 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:09,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:09,178 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:10,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:11,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:12,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:12,747 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:13,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:14,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:15,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:15,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:16,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:17,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:17,818 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:53:18,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:53:18,164 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:53:18,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:19,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:20,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:20,226 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:21,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:22,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:23,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:24,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:25,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:25,974 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:26,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:26,751 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:53:26,993 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/0qn0rtxn-events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:53:27,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:28,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:28,752 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:29,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:30,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:30,752 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:31,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:31,937 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:32,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:32,753 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:33,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:53:33,165 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:53:33,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:34,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:35,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:36,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:37,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:37,296 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:38,182 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:39,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:40,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:41,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:42,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:43,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:43,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:44,183 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:45,184 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:46,184 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:47,184 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:47,818 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:53:48,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:53:48,164 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:53:48,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:48,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:49,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:50,186 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:50,759 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:53:51,188 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:52,186 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:53,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:54,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:53:54,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:55,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:56,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:57,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:58,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:53:59,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:00,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:00,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:01,187 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:02,188 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:03,164 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:54:03,165 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:54:03,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:04,188 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:05,188 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:05,359 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:05,694 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:54:05,694 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 with policy now |
| 2025-01-07 16:54:05,703 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:54:05,704 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:54:05,704 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:54:05,704 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:54:05,764 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:54:05,764 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 |
| 2025-01-07 16:54:05,764 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval |
| 2025-01-07 16:54:05,952 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/i68rtlht-media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 |
| 2025-01-07 16:54:06,188 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:06,211 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/c0ynu488-media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 |
| 2025-01-07 16:54:06,764 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:07,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:08,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:09,201 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:10,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:10,766 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:11,199 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:11,201 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:12,192 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:13,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:14,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:14,767 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:15,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:16,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:17,170 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:17,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:17,819 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:54:18,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:54:18,166 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:54:18,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:19,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:20,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:21,193 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:22,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:22,278 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:23,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:24,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:24,770 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:54:25,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:25,773 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:54:25,774 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:54:25,775 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:54:26,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:26,775 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:26,776 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:54:27,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:27,775 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:28,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:29,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:30,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:30,777 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:31,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:32,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:32,777 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:33,097 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:33,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:54:33,165 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:54:33,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:34,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:35,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:36,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:37,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:38,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:38,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:39,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:40,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:41,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:42,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:43,171 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:43,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:44,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:45,196 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:46,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:47,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:47,819 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:54:48,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:54:48,165 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:54:48,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:48,289 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:49,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:50,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:51,197 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:52,213 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:52,784 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:53,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:54,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:54,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:54,785 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:54:55,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:56,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:57,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:58,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:54:59,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:54:59,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:00,199 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:01,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:02,199 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:03,173 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:55:03,173 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:55:03,200 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:04,214 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:04,225 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:04,272 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:55:04,272 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 with policy now |
| 2025-01-07 16:55:04,277 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:55:04,278 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:55:04,278 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:55:04,289 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:55:04,530 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/bp0cxicr-media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 |
| 2025-01-07 16:55:04,795 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:04,795 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:55:04,795 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 |
| 2025-01-07 16:55:04,795 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval |
| 2025-01-07 16:55:05,066 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/22ro6c6m-media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 |
| 2025-01-07 16:55:05,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:06,213 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:06,795 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:07,215 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:08,205 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:08,796 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:09,207 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:10,174 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:10,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:11,212 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:12,212 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:12,797 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:13,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:14,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:15,207 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:15,531 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:55:15,531 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:55:15,532 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:55:15,532 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:15,799 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:55:15,799 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:55:16,067 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/thkxy9jr-events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:55:16,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:16,799 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:17,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:17,820 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:55:18,168 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:55:18,169 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:55:18,220 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:19,206 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:20,207 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:20,800 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:21,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:21,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:22,207 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:23,207 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:24,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:25,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:26,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:27,175 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:27,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:28,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:29,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:30,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:31,208 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:32,175 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:32,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:33,165 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:55:33,166 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:55:33,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:34,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:35,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:36,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:37,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:37,247 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:38,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:39,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:40,209 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:41,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:42,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:43,175 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:43,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:44,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:45,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:46,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:47,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:47,820 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:55:48,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:55:48,166 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:55:48,213 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:48,246 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:49,210 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:50,211 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:51,211 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:52,211 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:53,211 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:53,722 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:55:54,212 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:54,812 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:54,995 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:55:54,995 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 with policy now |
| 2025-01-07 16:55:54,998 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:55:54,998 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:55:54,998 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:55:55,006 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:55:55,213 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:55,267 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/7d38p6ic-media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 |
| 2025-01-07 16:55:55,812 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:55:55,812 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 |
| 2025-01-07 16:55:55,813 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval |
| 2025-01-07 16:55:56,112 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/uu8xueua-media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 |
| 2025-01-07 16:55:56,222 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:56,812 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:55:57,217 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:58,221 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:59,220 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:55:59,689 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:00,221 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:00,814 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:01,222 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:02,224 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:02,814 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:03,169 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:56:03,169 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:56:03,226 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:04,224 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:05,226 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:05,278 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:06,229 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:07,226 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:08,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:09,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:10,228 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:11,180 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:11,229 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:12,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:13,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:13,604 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:56:13,605 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:56:13,606 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:56:13,818 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:56:13,818 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:56:14,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:14,819 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:15,233 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:16,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:16,743 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:17,233 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:17,820 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:56:18,167 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:56:18,167 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:56:18,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:18,820 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:19,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:20,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:21,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:22,227 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:22,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:23,234 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:24,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:25,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:26,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:27,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:27,565 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:28,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:28,823 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:29,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:30,235 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:31,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:32,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:33,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:56:33,167 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:56:33,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:33,253 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:34,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:34,825 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:35,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:36,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:37,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:38,236 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:39,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:39,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:40,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:41,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:42,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:43,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:44,179 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:44,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:45,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:46,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:47,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:47,821 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:56:48,166 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:56:48,167 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:56:48,237 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:49,238 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:49,239 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:50,238 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:51,238 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:52,240 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:52,832 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:53,253 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:53,281 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:56:53,281 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 with policy now |
| 2025-01-07 16:56:53,325 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:56:53,325 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:56:53,325 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:56:53,334 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:56:53,578 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/cwb1a6r0-media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 |
| 2025-01-07 16:56:53,832 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:56:53,832 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 |
| 2025-01-07 16:56:53,832 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval |
| 2025-01-07 16:56:54,114 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/3rpgmo3u-media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 |
| 2025-01-07 16:56:54,244 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:54,376 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:56:54,832 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:55,240 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:55,833 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:56,243 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:57,248 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:58,245 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:58,834 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:56:59,244 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:56:59,575 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:00,248 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:00,834 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:57:01,249 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:02,251 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:03,176 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:57:03,176 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:57:03,252 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:04,254 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:05,253 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:05,259 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:06,261 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:07,253 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:08,254 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:09,258 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:10,259 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:11,198 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:11,260 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:12,260 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:13,262 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:14,263 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:15,264 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:16,199 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:16,268 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:17,267 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:17,822 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:57:18,174 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:57:18,174 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:57:18,270 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:19,272 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:20,269 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:21,281 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:21,291 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:22,290 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:23,276 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:24,290 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:25,286 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:26,291 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:27,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:27,283 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:28,293 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:29,285 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:30,286 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:31,311 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:32,195 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:32,294 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:32,702 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:57:32,702 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:32,703 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:32,845 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:57:32,845 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:57:33,090 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/jr8j7op1-events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:57:33,181 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 16:57:33,181 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 16:57:33,267 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 16:57:33,280 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 16:57:33,304 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:33,482 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:57:33,693 INFO wandb-upload_0:2190370 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp3pg3budj |
| 2025-01-07 16:57:33,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:57:33,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-07 16:57:33,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media |
| 2025-01-07 16:57:33,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table |
| 2025-01-07 16:57:33,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval |
| 2025-01-07 16:57:33,909 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 16:57:33,922 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 16:57:34,085 INFO SenderThread:2190370 [sender.py:send_request_log_artifact():1525] logged artifact run-nuw4e53j-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTg0NzQ3Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY3ODk2MQ==', 'latestArtifact': None}} |
| 2025-01-07 16:57:34,086 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:57:34,086 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json with policy now |
| 2025-01-07 16:57:34,086 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:57:34,086 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,086 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,087 DEBUG SenderThread:2190370 [sender.py:send():379] send: telemetry |
| 2025-01-07 16:57:34,087 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,087 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,087 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,087 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,088 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,088 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,088 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,088 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,089 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,089 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,089 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,090 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,090 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,090 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,090 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,090 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,091 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,091 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,091 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,091 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,092 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,092 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,092 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 16:57:34,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:34,291 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/8k76rw4h-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-07 16:57:34,447 INFO wandb-upload_0:2190370 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpabm6uday |
| 2025-01-07 16:57:34,537 INFO wandb-upload_2:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/od0q09ti-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-07 16:57:34,807 INFO SenderThread:2190370 [sender.py:send_request_log_artifact():1525] logged artifact run-nuw4e53j-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTg0NzUyOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY3ODk2NA==', 'latestArtifact': None}} |
| 2025-01-07 16:57:34,807 DEBUG SenderThread:2190370 [sender.py:send():379] send: files |
| 2025-01-07 16:57:34,807 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json with policy now |
| 2025-01-07 16:57:34,807 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:57:34,807 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:34,808 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:34,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:57:34,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:57:34,846 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json |
| 2025-01-07 16:57:34,847 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval |
| 2025-01-07 16:57:35,007 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/0utqlxre-media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json |
| 2025-01-07 16:57:35,277 INFO wandb-upload_2:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/qserdyai-media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json |
| 2025-01-07 16:57:35,287 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:36,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:37,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:37,808 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:38,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:39,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:40,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:41,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:42,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:43,194 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:43,288 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:43,869 DEBUG SenderThread:2190370 [sender.py:send():379] send: exit |
| 2025-01-07 16:57:43,869 INFO SenderThread:2190370 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 16:57:43,869 INFO SenderThread:2190370 [sender.py:send_exit():588] handling runtime: 356 |
| 2025-01-07 16:57:43,870 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:43,870 INFO SenderThread:2190370 [sender.py:send_exit():594] send defer |
| 2025-01-07 16:57:43,870 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:43,870 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 16:57:43,870 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:43,870 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 16:57:43,870 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 16:57:43,870 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:43,870 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 16:57:43,871 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:43,871 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 16:57:43,871 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 16:57:43,871 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:43,871 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 16:57:43,871 INFO HandlerThread:2190370 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 16:57:43,871 DEBUG SystemMonitor:2190370 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 16:57:43,871 DEBUG SystemMonitor:2190370 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 16:57:43,872 INFO HandlerThread:2190370 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 16:57:43,872 INFO HandlerThread:2190370 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 16:57:43,925 INFO HandlerThread:2190370 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 16:57:43,926 INFO HandlerThread:2190370 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 16:57:43,926 INFO HandlerThread:2190370 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 16:57:43,926 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:43,926 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 16:57:43,926 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 16:57:43,926 DEBUG SenderThread:2190370 [sender.py:send():379] send: stats |
| 2025-01-07 16:57:43,927 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:43,927 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 16:57:43,927 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:43,927 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 16:57:43,927 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 16:57:43,927 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:43,927 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 16:57:44,850 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:57:44,851 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:57:49,852 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/config.yaml |
| 2025-01-07 16:57:50,707 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 16:57:50,707 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:50,707 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 16:57:50,708 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:50,708 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 16:57:50,708 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 16:57:50,708 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 16:57:50,709 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:50,709 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 16:57:50,709 DEBUG SenderThread:2190370 [sender.py:send():379] send: history |
| 2025-01-07 16:57:50,709 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 16:57:50,710 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:50,711 DEBUG SenderThread:2190370 [sender.py:send():379] send: summary |
| 2025-01-07 16:57:50,711 INFO SenderThread:2190370 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 16:57:50,711 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:50,711 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 16:57:50,711 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 16:57:50,711 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:50,711 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 16:57:50,712 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:50,712 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 16:57:50,712 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 16:57:50,712 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 16:57:50,712 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:50,712 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 16:57:50,712 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:50,712 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 16:57:50,853 INFO Thread-12 :2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:57:50,870 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 16:57:53,225 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 16:57:53,225 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 16:57:53,225 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:53,226 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 16:57:53,226 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:53,226 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 16:57:53,226 INFO SenderThread:2190370 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 16:57:53,231 INFO SenderThread:2190370 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 16:57:53,234 INFO SenderThread:2190370 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 16:57:53,243 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 16:57:53,243 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:53,243 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 16:57:53,243 DEBUG SenderThread:2190370 [sender.py:send():379] send: artifact |
| 2025-01-07 16:57:53,440 INFO SenderThread:2190370 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTgyOTA5NQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTgyOTA5NQ==', 'versionIndex': 433}}} |
| 2025-01-07 16:57:53,441 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:53,441 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 16:57:53,441 INFO SenderThread:2190370 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 16:57:53,854 INFO SenderThread:2190370 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:57:53,854 INFO SenderThread:2190370 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files |
| 2025-01-07 16:57:53,855 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/config.yaml config.yaml |
| 2025-01-07 16:57:53,855 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 16:57:53,855 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:57:53,855 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/requirements.txt requirements.txt |
| 2025-01-07 16:57:53,859 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 16:57:53,860 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log output.log |
| 2025-01-07 16:57:53,860 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/diff.patch diff.patch |
| 2025-01-07 16:57:53,860 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 16:57:53,862 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 media/videos/eval/2_eval_2_1ac5ce9261e915f80fbf.mp4 |
| 2025-01-07 16:57:53,862 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 media/videos/eval/4_eval_4_093e982469739dd004fa.mp4 |
| 2025-01-07 16:57:53,862 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 media/videos/eval/0_eval_0_f55a79f6e13db762147d.mp4 |
| 2025-01-07 16:57:53,862 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 media/videos/eval/3_eval_3_225e281ad921a8071d1b.mp4 |
| 2025-01-07 16:57:53,862 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 media/videos/eval/1_eval_1_f2c601bce116aeb8bef1.mp4 |
| 2025-01-07 16:57:53,867 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json media/table/eval/total_success_rate_ci_6_9eab7a1bd48547a874a3.table.json |
| 2025-01-07 16:57:53,867 INFO SenderThread:2190370 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-07 16:57:53,867 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 16:57:53,867 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:53,868 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 16:57:53,868 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:53,868 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 16:57:53,868 INFO SenderThread:2190370 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 16:57:53,871 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 16:57:53,872 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 16:57:54,041 INFO wandb-upload_0:2190370 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/wandb-summary.json |
| 2025-01-07 16:57:54,057 INFO wandb-upload_1:2190370 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/config.yaml |
| 2025-01-07 16:57:54,062 INFO wandb-upload_3:2190370 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/requirements.txt |
| 2025-01-07 16:57:54,092 INFO wandb-upload_2:2190370 [upload_job.py:push():130] Uploaded file /tmp/tmpzlsj7wjwwandb/f693zx7u-events.out.tfevents.1736286708.leonmkim-ROG-Strix-G15CS-G15CS.2190176.0 |
| 2025-01-07 16:57:54,116 INFO wandb-upload_4:2190370 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/files/output.log |
| 2025-01-07 16:57:54,317 INFO Thread-11 (_thread_body):2190370 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 16:57:54,317 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:54,317 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 16:57:54,317 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:54,317 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 16:57:54,317 INFO SenderThread:2190370 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 16:57:54,318 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 16:57:54,319 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:54,319 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 16:57:54,319 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:54,319 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 16:57:54,319 INFO SenderThread:2190370 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 16:57:54,507 INFO SenderThread:2190370 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 16:57:54,507 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 16:57:54,508 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:54,508 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 16:57:54,508 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:54,508 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 16:57:54,508 INFO SenderThread:2190370 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 16:57:54,508 DEBUG SenderThread:2190370 [sender.py:send():379] send: final |
| 2025-01-07 16:57:54,508 DEBUG SenderThread:2190370 [sender.py:send():379] send: footer |
| 2025-01-07 16:57:54,508 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 16:57:54,508 INFO HandlerThread:2190370 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 16:57:54,508 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 16:57:54,509 INFO SenderThread:2190370 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 16:57:54,509 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 16:57:54,509 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 16:57:54,509 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 16:57:54,510 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 16:57:54,510 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 16:57:54,511 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 16:57:54,512 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 16:57:54,512 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 16:57:54,562 DEBUG SenderThread:2190370 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 16:57:54,562 INFO MainThread:2190370 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 16:57:54,563 INFO MainThread:2190370 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 16:57:54,563 INFO MainThread:2190370 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 16:57:54,563 DEBUG HandlerThread:2190370 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 16:57:54,563 INFO HandlerThread:2190370 [handler.py:finish():882] shutting down handler |
| 2025-01-07 16:57:55,510 INFO WriterThread:2190370 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/nuw4e53j/wandb/run-20250107_165147-nuw4e53j/run-nuw4e53j.wandb |
| 2025-01-07 16:57:55,562 INFO SenderThread:2190370 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 16:57:55,562 INFO SenderThread:2190370 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 16:57:55,562 INFO SenderThread:2190370 [file_pusher.py:join():175] waiting for file pusher |
|
|
