| 2025-01-07 18:06:19,355 INFO StreamThr :2207962 [internal.py:wandb_internal():85] W&B internal server running at pid: 2207962, started at: 2025-01-07 18:06:19.355271 |
| 2025-01-07 18:06:19,356 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 18:06:19,368 INFO WriterThread:2207962 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/run-hcdwvdao.wandb |
| 2025-01-07 18:06:19,368 DEBUG SenderThread:2207962 [sender.py:send():379] send: header |
| 2025-01-07 18:06:19,375 DEBUG SenderThread:2207962 [sender.py:send():379] send: run |
| 2025-01-07 18:06:19,630 INFO SenderThread:2207962 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files |
| 2025-01-07 18:06:19,631 INFO SenderThread:2207962 [sender.py:_start_run_threads():1188] run started: hcdwvdao with start time 1736291179.357482 |
| 2025-01-07 18:06:19,646 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 18:06:19,647 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 18:06:19,720 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 18:06:19,725 DEBUG HandlerThread:2207962 [system_info.py:__init__():26] System info init |
| 2025-01-07 18:06:19,725 DEBUG HandlerThread:2207962 [system_info.py:__init__():41] System info init done |
| 2025-01-07 18:06:19,725 INFO HandlerThread:2207962 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 18:06:19,725 INFO SystemMonitor:2207962 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 18:06:19,725 INFO HandlerThread:2207962 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 18:06:19,726 INFO SystemMonitor:2207962 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 18:06:19,727 INFO SystemMonitor:2207962 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 18:06:19,727 INFO SystemMonitor:2207962 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 18:06:19,728 INFO SystemMonitor:2207962 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 18:06:19,728 INFO SystemMonitor:2207962 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 18:06:19,739 DEBUG HandlerThread:2207962 [system_info.py:probe():152] Probing system |
| 2025-01-07 18:06:19,740 DEBUG HandlerThread:2207962 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 18:06:19,745 DEBUG HandlerThread:2207962 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 18:06:19,745 DEBUG HandlerThread:2207962 [system_info.py:probe():200] Probing system done |
| 2025-01-07 18:06:19,745 DEBUG HandlerThread:2207962 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T23:06:19.739320', 'startedAt': '2025-01-07T23:06:19.347690', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3566.71675, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4502.251, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4509.34, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 802.8848648071289}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 18:06:19,746 INFO HandlerThread:2207962 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 18:06:19,746 INFO HandlerThread:2207962 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 18:06:19,746 DEBUG HandlerThread:2207962 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 18:06:19,747 DEBUG HandlerThread:2207962 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 18:06:19,747 DEBUG HandlerThread:2207962 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 18:06:19,809 DEBUG HandlerThread:2207962 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 18:06:19,810 INFO HandlerThread:2207962 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 18:06:19,813 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:06:19,813 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 18:06:19,813 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 18:06:19,813 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 18:06:20,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 18:06:20,054 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 18:06:20,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:06:20,055 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:06:20,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:20,057 INFO HandlerThread:2207962 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/tb" |
| save: true |
| } |
| control { |
| relay_id: "55d9574faf0842428b21c40159837d41" |
| } |
| _info { |
| stream_id: "hcdwvdao" |
| } |
|
|
| 2025-01-07 18:06:20,057 DEBUG HandlerThread:2207962 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 18:06:20,068 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/eifv9h5e-code/FISH/eval_robot.py |
| 2025-01-07 18:06:20,082 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/3cvz4p6z-wandb-metadata.json |
| 2025-01-07 18:06:20,132 DEBUG SenderThread:2207962 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:06:20,132 DEBUG SenderThread:2207962 [sender.py:send():379] send: tbrecord |
| 2025-01-07 18:06:20,132 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:06:20,133 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 with policy live |
| 2025-01-07 18:06:20,142 INFO wandb-upload_2:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/3i0p729g-diff.patch |
| 2025-01-07 18:06:20,321 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/h3heor1y-events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:06:20,632 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:20,632 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/requirements.txt |
| 2025-01-07 18:06:20,632 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/code/FISH/eval_robot.py |
| 2025-01-07 18:06:20,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-metadata.json |
| 2025-01-07 18:06:20,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/diff.patch |
| 2025-01-07 18:06:20,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:06:20,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/code/FISH |
| 2025-01-07 18:06:20,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/code |
| 2025-01-07 18:06:21,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:22,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:22,631 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:23,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:23,772 DEBUG SenderThread:2207962 [sender.py:send():379] send: config |
| 2025-01-07 18:06:24,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:24,785 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:25,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:26,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:26,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:27,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:28,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:28,633 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:29,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:30,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:30,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:30,634 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:31,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:32,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:32,634 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:33,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:34,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:35,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:06:35,054 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:06:35,097 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:35,115 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:36,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:37,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:38,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:39,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:40,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:40,637 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:41,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:41,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:42,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:42,637 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:43,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:44,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:44,638 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:06:45,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:46,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:46,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:47,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:48,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:49,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:50,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:06:50,054 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:06:50,101 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:51,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:51,169 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:51,640 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/config.yaml |
| 2025-01-07 18:06:52,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:53,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:54,060 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:55,060 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:56,060 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:57,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:06:57,060 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:58,060 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:06:59,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:00,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:01,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:02,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:02,497 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:02,642 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:03,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:04,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:04,643 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:05,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:07:05,054 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:07:05,097 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:06,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:07,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:08,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:08,153 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:09,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:10,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:11,063 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:12,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:13,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:14,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:14,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:15,063 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:16,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:17,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:17,802 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:07:17,802 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 with policy now |
| 2025-01-07 18:07:17,803 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:07:17,806 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:07:17,806 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:07:17,807 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:07:18,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:18,067 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/xfkr53fc-media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 |
| 2025-01-07 18:07:18,647 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:18,647 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 |
| 2025-01-07 18:07:18,647 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:07:18,647 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media |
| 2025-01-07 18:07:18,648 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos |
| 2025-01-07 18:07:18,648 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval |
| 2025-01-07 18:07:18,948 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/xmoxyftm-media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 |
| 2025-01-07 18:07:19,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:19,728 DEBUG SystemMonitor:2207962 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 18:07:19,729 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:07:19,729 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:20,054 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:07:20,054 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:07:20,097 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:20,647 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:21,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:22,064 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:22,648 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:23,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:24,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:25,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:25,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:26,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:26,649 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:27,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:28,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:28,650 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:07:28,945 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/mwkiq0be-events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:07:29,066 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:30,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:30,651 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:31,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:31,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:32,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:33,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:34,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:34,652 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:07:35,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:07:35,059 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:07:35,101 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:36,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:36,218 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:37,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:38,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:39,068 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:40,067 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:41,068 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:42,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:42,068 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:43,068 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:44,068 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:45,069 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:46,069 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:47,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:47,069 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:48,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:49,069 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:49,729 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:07:50,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:07:50,055 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:07:50,097 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:51,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:52,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:52,214 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:53,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:54,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:55,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:56,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:57,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:58,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:07:58,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:07:59,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:00,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:01,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:02,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:03,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:03,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:04,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:05,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:08:05,055 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:08:05,101 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:06,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:07,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:08,073 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:08,423 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:08,663 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:08,676 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:08:08,676 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 with policy now |
| 2025-01-07 18:08:08,679 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:08:08,679 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:08:08,679 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:08:08,683 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:08:08,951 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/ejiokp1q-media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 |
| 2025-01-07 18:08:09,073 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:09,663 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:08:09,663 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 |
| 2025-01-07 18:08:09,663 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval |
| 2025-01-07 18:08:09,984 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/qmp57b2j-media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 |
| 2025-01-07 18:08:10,075 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:10,663 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:11,078 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:12,080 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:13,080 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:13,453 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:14,078 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:14,665 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:15,080 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:16,079 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:16,665 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:17,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:18,088 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:18,666 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:08:19,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:19,413 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:19,494 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:08:19,494 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:08:19,495 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:08:19,667 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:08:19,730 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:08:20,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:08:20,057 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:08:20,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:20,667 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:21,086 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:22,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:23,082 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:24,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:24,483 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:24,668 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:25,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:26,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:27,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:28,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:29,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:30,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:30,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:31,083 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:32,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:33,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:34,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:35,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:08:35,055 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:08:35,097 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:35,149 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:36,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:37,084 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:38,277 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:39,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:40,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:41,061 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:41,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:42,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:42,680 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:08:43,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:44,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:45,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:46,063 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:46,271 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:47,272 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:48,272 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:49,272 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:49,730 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:08:50,055 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:08:50,056 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:08:50,272 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:51,114 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:51,272 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:52,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:53,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:54,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:55,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:56,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:57,062 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:08:57,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:58,273 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:59,217 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:08:59,217 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 with policy now |
| 2025-01-07 18:08:59,220 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:08:59,220 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:08:59,220 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:08:59,226 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:08:59,274 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:08:59,473 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/400suiaq-media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 |
| 2025-01-07 18:08:59,687 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:08:59,688 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 |
| 2025-01-07 18:08:59,688 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval |
| 2025-01-07 18:08:59,964 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/7gfiypmj-media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 |
| 2025-01-07 18:09:00,281 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:00,688 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:09:01,276 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:02,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:02,278 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:03,278 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:04,278 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:04,689 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:09:05,058 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:09:05,058 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:09:05,277 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:06,278 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:06,690 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:09:07,277 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:07,777 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:08,279 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:09,282 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:10,286 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:11,282 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:12,284 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:13,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:13,291 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:14,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:15,287 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:16,289 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:17,290 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:18,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:18,290 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:18,347 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:09:18,347 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:09:18,348 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:09:18,694 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:09:18,694 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:09:18,955 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/neeeybwj-events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:09:19,321 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:19,731 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:09:20,057 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:09:20,057 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:09:20,291 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:20,694 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:09:21,295 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:22,287 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:23,280 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:23,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:24,287 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:24,696 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:09:25,287 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:26,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:27,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:28,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:29,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:29,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:30,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:31,288 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:32,289 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:33,290 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:34,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:34,291 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:35,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:09:35,057 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:09:35,290 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:36,290 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:37,291 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:38,291 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:39,118 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:39,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:40,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:41,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:42,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:43,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:44,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:45,070 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:45,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:46,292 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:47,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:48,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:49,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:49,731 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:09:50,056 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:09:50,056 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:09:50,211 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:50,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:51,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:52,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:53,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:54,293 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:55,294 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:56,071 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:09:56,294 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:57,295 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:58,295 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:58,999 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:09:58,999 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 with policy now |
| 2025-01-07 18:09:59,002 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:09:59,002 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:09:59,003 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:09:59,009 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:09:59,262 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/5ymy2z1j-media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 |
| 2025-01-07 18:09:59,296 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:09:59,708 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:09:59,708 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 |
| 2025-01-07 18:09:59,708 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval |
| 2025-01-07 18:09:59,989 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/6l876urn-media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 |
| 2025-01-07 18:10:00,302 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:00,708 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:01,075 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:01,300 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:02,299 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:03,301 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:04,306 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:04,709 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:05,063 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:10:05,063 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:10:05,314 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:06,303 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:06,573 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:06,710 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:07,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:08,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:09,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:09,711 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:10:10,302 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:10,412 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:10:10,413 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:10:10,413 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:10:10,711 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:10:10,712 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:11,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:11,693 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:12,309 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:12,712 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:13,306 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:14,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:14,713 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:15,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:16,304 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:16,714 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:17,072 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:17,305 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:18,305 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:19,305 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:19,732 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:10:20,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:10:20,059 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:10:20,305 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:21,307 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:22,170 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:22,306 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:23,306 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:24,307 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:25,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:26,309 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:27,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:28,073 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:28,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:29,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:30,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:31,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:32,308 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:33,073 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:33,309 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:34,309 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:35,059 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:10:35,059 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:10:35,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:36,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:37,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:38,143 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:38,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:39,310 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:40,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:41,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:42,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:43,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:44,073 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:44,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:45,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:46,311 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:47,312 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:48,312 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:49,312 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:49,391 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:49,732 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:10:50,074 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:10:50,074 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:10:50,322 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:50,429 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:10:50,429 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 with policy now |
| 2025-01-07 18:10:50,435 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:10:50,435 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:10:50,435 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:10:50,443 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:10:50,727 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:10:50,727 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:50,727 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 |
| 2025-01-07 18:10:50,727 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval |
| 2025-01-07 18:10:50,737 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/ioczzw4e-media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 |
| 2025-01-07 18:10:51,009 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/dmn7kbli-media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 |
| 2025-01-07 18:10:51,324 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:52,331 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:52,727 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:53,324 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:54,327 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:55,079 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:10:55,326 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:56,330 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:56,729 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:57,332 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:58,329 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:10:58,729 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:10:59,329 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:00,090 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:11:00,336 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:01,338 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:02,333 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:03,338 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:04,340 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:05,065 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:11:05,065 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:11:05,189 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:11:05,342 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:06,341 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:07,336 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:08,340 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:09,348 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:09,490 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:11:09,490 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:09,490 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:09,733 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:11:09,733 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:11:09,994 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/xj7zdd1l-events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:11:10,313 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:11:10,313 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:11:10,357 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:10,553 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:11:10,733 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:11:10,733 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:11:10,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media |
| 2025-01-07 18:11:10,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval |
| 2025-01-07 18:11:10,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table |
| 2025-01-07 18:11:10,820 INFO wandb-upload_1:2207962 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpn0uzkz_a |
| 2025-01-07 18:11:10,966 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:11:10,980 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:11:10,990 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/dc8zpfxj-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:11:11,194 INFO SenderThread:2207962 [sender.py:send_request_log_artifact():1525] logged artifact run-hcdwvdao-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjA2NjkyMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTc0MzgzNQ==', 'latestArtifact': None}} |
| 2025-01-07 18:11:11,194 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:11:11,194 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-07 18:11:11,195 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:11:11,195 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:11:11,195 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,196 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,196 DEBUG SenderThread:2207962 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:11:11,196 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,200 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,200 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,200 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,200 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,200 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,201 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,201 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,201 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,202 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,202 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,202 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,202 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,203 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,203 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,203 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,203 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,204 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,204 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,204 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,204 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,205 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,205 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:11:11,338 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:11,466 INFO wandb-upload_2:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/gq1xcm8x-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:11:11,587 INFO wandb-upload_0:2207962 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpa5g1zu2x |
| 2025-01-07 18:11:11,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:11:11,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:11:11,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json |
| 2025-01-07 18:11:11,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval |
| 2025-01-07 18:11:11,962 INFO SenderThread:2207962 [sender.py:send_request_log_artifact():1525] logged artifact run-hcdwvdao-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjA2NjkzMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTc0MzgzNg==', 'latestArtifact': None}} |
| 2025-01-07 18:11:11,962 DEBUG SenderThread:2207962 [sender.py:send():379] send: files |
| 2025-01-07 18:11:11,962 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json with policy now |
| 2025-01-07 18:11:11,962 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:11:11,963 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:11,963 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,963 DEBUG SenderThread:2207962 [sender.py:send():379] send: exit |
| 2025-01-07 18:11:11,963 INFO SenderThread:2207962 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 18:11:11,963 INFO SenderThread:2207962 [sender.py:send_exit():588] handling runtime: 291 |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:send_exit():594] send defer |
| 2025-01-07 18:11:11,964 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:11,964 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 18:11:11,964 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 18:11:11,964 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:11,964 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 18:11:11,964 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 18:11:11,964 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 18:11:11,965 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:11,965 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 18:11:11,965 INFO HandlerThread:2207962 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 18:11:11,965 INFO HandlerThread:2207962 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 18:11:11,965 DEBUG SystemMonitor:2207962 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 18:11:11,965 INFO HandlerThread:2207962 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 18:11:11,965 DEBUG SystemMonitor:2207962 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 18:11:12,017 INFO HandlerThread:2207962 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 18:11:12,018 INFO HandlerThread:2207962 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 18:11:12,018 INFO HandlerThread:2207962 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 18:11:12,018 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:12,018 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 18:11:12,018 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 18:11:12,018 DEBUG SenderThread:2207962 [sender.py:send():379] send: stats |
| 2025-01-07 18:11:12,019 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:12,019 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 18:11:12,019 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:12,019 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 18:11:12,019 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 18:11:12,019 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:12,019 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 18:11:12,147 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/pta6s1go-media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json |
| 2025-01-07 18:11:12,404 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/rmqd7loz-media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json |
| 2025-01-07 18:11:12,734 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:11:12,735 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:11:14,735 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:11:18,492 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:18,492 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:11:18,492 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 18:11:18,493 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:18,493 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 18:11:18,493 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 18:11:18,494 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:18,494 DEBUG SenderThread:2207962 [sender.py:send():379] send: history |
| 2025-01-07 18:11:18,494 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:11:18,494 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:18,495 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 18:11:18,496 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:18,496 DEBUG SenderThread:2207962 [sender.py:send():379] send: summary |
| 2025-01-07 18:11:18,500 INFO SenderThread:2207962 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:11:18,500 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:18,500 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 18:11:18,500 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 18:11:18,500 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:18,501 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 18:11:18,501 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:18,501 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 18:11:18,504 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:11:18,633 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 18:11:18,633 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:18,633 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 18:11:18,634 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:18,634 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 18:11:18,737 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:11:18,737 INFO Thread-12 :2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/config.yaml |
| 2025-01-07 18:11:19,459 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:20,425 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 18:11:20,425 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:20,425 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:20,425 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 18:11:20,426 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:20,426 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 18:11:20,426 INFO SenderThread:2207962 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 18:11:20,428 INFO SenderThread:2207962 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 18:11:20,430 INFO SenderThread:2207962 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 18:11:20,445 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 18:11:20,445 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:20,445 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 18:11:20,445 DEBUG SenderThread:2207962 [sender.py:send():379] send: artifact |
| 2025-01-07 18:11:20,459 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:20,670 INFO SenderThread:2207962 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTgyOTA5NQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'versionIndex': 436}}} |
| 2025-01-07 18:11:20,671 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:20,671 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 18:11:20,671 INFO SenderThread:2207962 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 18:11:20,738 INFO SenderThread:2207962 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:11:20,738 INFO SenderThread:2207962 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files |
| 2025-01-07 18:11:20,738 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/config.yaml config.yaml |
| 2025-01-07 18:11:20,739 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 18:11:20,740 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:11:20,740 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/requirements.txt requirements.txt |
| 2025-01-07 18:11:20,740 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 18:11:20,741 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log output.log |
| 2025-01-07 18:11:20,741 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/diff.patch diff.patch |
| 2025-01-07 18:11:20,742 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 18:11:20,742 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 media/videos/eval/1_eval_1_d770d2c2408f04d7abb1.mp4 |
| 2025-01-07 18:11:20,742 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 media/videos/eval/0_eval_0_b0912050a1ad9dbe66ae.mp4 |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 media/videos/eval/4_eval_4_2805344fc05b94c9bd94.mp4 |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 media/videos/eval/3_eval_3_b4b0a7d2f5d2484c5d27.mp4 |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 media/videos/eval/2_eval_2_a9b10b1d7fc921011b04.mp4 |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json media/table/eval/total_success_rate_ci_6_add3878bc9cb899afed4.table.json |
| 2025-01-07 18:11:20,743 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 18:11:20,743 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:20,746 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:20,746 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 18:11:20,748 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:20,748 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 18:11:20,748 INFO SenderThread:2207962 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:11:20,933 INFO wandb-upload_1:2207962 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/config.yaml |
| 2025-01-07 18:11:20,934 INFO wandb-upload_0:2207962 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/wandb-summary.json |
| 2025-01-07 18:11:20,984 INFO wandb-upload_4:2207962 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/output.log |
| 2025-01-07 18:11:20,984 INFO wandb-upload_3:2207962 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/files/requirements.txt |
| 2025-01-07 18:11:21,035 INFO wandb-upload_2:2207962 [upload_job.py:push():130] Uploaded file /tmp/tmpo5oxjgeowandb/n4hex9g0-events.out.tfevents.1736291180.leonmkim-ROG-Strix-G15CS-G15CS.2207769.0 |
| 2025-01-07 18:11:21,235 INFO Thread-11 (_thread_body):2207962 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 18:11:21,235 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:21,235 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 18:11:21,236 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:21,236 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 18:11:21,236 INFO SenderThread:2207962 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 18:11:21,237 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 18:11:21,237 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:21,237 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 18:11:21,237 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:21,237 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 18:11:21,237 INFO SenderThread:2207962 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 18:11:21,460 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:21,561 INFO SenderThread:2207962 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 18:11:21,561 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 18:11:21,561 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:21,561 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:21,561 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 18:11:21,561 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:21,562 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 18:11:21,562 INFO SenderThread:2207962 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 18:11:21,562 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:11:21,562 INFO HandlerThread:2207962 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 18:11:21,562 DEBUG SenderThread:2207962 [sender.py:send():379] send: final |
| 2025-01-07 18:11:21,562 DEBUG SenderThread:2207962 [sender.py:send():379] send: footer |
| 2025-01-07 18:11:21,563 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:11:21,563 INFO SenderThread:2207962 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 18:11:21,563 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:21,563 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:21,564 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 18:11:21,564 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:11:21,564 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 18:11:21,565 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 18:11:21,565 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 18:11:21,565 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:11:21,618 DEBUG SenderThread:2207962 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:11:21,618 INFO MainThread:2207962 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 18:11:21,619 INFO MainThread:2207962 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 18:11:21,619 INFO MainThread:2207962 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 18:11:21,619 DEBUG HandlerThread:2207962 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 18:11:21,619 INFO HandlerThread:2207962 [handler.py:finish():882] shutting down handler |
| 2025-01-07 18:11:22,564 INFO WriterThread:2207962 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/hcdwvdao/wandb/run-20250107_180619-hcdwvdao/run-hcdwvdao.wandb |
| 2025-01-07 18:11:22,618 INFO SenderThread:2207962 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 18:11:22,618 INFO SenderThread:2207962 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:11:22,618 INFO SenderThread:2207962 [file_pusher.py:join():175] waiting for file pusher |
|
|
