| 2024-12-10 21:40:38,452 INFO StreamThr :608881 [internal.py:wandb_internal():85] W&B internal server running at pid: 608881, started at: 2024-12-10 21:40:38.452460 |
| 2024-12-10 21:40:38,453 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status |
| 2024-12-10 21:40:38,464 INFO WriterThread:608881 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/run-s9hpsjf9.wandb |
| 2024-12-10 21:40:38,465 DEBUG SenderThread:608881 [sender.py:send():379] send: header |
| 2024-12-10 21:40:38,473 DEBUG SenderThread:608881 [sender.py:send():379] send: run |
| 2024-12-10 21:40:38,722 INFO SenderThread:608881 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files |
| 2024-12-10 21:40:38,722 INFO SenderThread:608881 [sender.py:_start_run_threads():1188] run started: s9hpsjf9 with start time 1733884838.452824 |
| 2024-12-10 21:40:38,737 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-10 21:40:38,737 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: check_version |
| 2024-12-10 21:40:38,787 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-10 21:40:38,792 DEBUG HandlerThread:608881 [system_info.py:__init__():26] System info init |
| 2024-12-10 21:40:38,792 DEBUG HandlerThread:608881 [system_info.py:__init__():41] System info init done |
| 2024-12-10 21:40:38,792 INFO HandlerThread:608881 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-10 21:40:38,792 INFO SystemMonitor:608881 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-10 21:40:38,792 INFO HandlerThread:608881 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-10 21:40:38,793 INFO SystemMonitor:608881 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-10 21:40:38,793 INFO SystemMonitor:608881 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-10 21:40:38,794 INFO SystemMonitor:608881 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-10 21:40:38,794 INFO SystemMonitor:608881 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-10 21:40:38,795 INFO SystemMonitor:608881 [interfaces.py:start():188] Started network monitoring |
| 2024-12-10 21:40:38,805 DEBUG HandlerThread:608881 [system_info.py:probe():152] Probing system |
| 2024-12-10 21:40:38,807 DEBUG HandlerThread:608881 [system_info.py:_probe_git():137] Probing git |
| 2024-12-10 21:40:38,811 DEBUG HandlerThread:608881 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-10 21:40:38,811 DEBUG HandlerThread:608881 [system_info.py:probe():200] Probing system done |
| 2024-12-10 21:40:38,811 DEBUG HandlerThread:608881 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-11T02:40:38.805785', 'startedAt': '2024-12-11T02:40:38.443244', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '4f7c0421eb84a7caf6bf5670f68777523362676f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3754.252, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4503.722, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 402.6240234375}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71596908569336}} |
| 2024-12-10 21:40:38,812 INFO HandlerThread:608881 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-10 21:40:38,812 INFO HandlerThread:608881 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-10 21:40:38,812 DEBUG HandlerThread:608881 [system_info.py:_save_code():44] Saving code |
| 2024-12-10 21:40:38,813 DEBUG HandlerThread:608881 [system_info.py:_save_code():67] Saving code done |
| 2024-12-10 21:40:38,813 DEBUG HandlerThread:608881 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-10 21:40:38,875 DEBUG HandlerThread:608881 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-10 21:40:38,875 INFO HandlerThread:608881 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-10 21:40:38,878 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:40:38,878 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-10 21:40:38,878 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-10 21:40:38,879 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-10 21:40:39,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-10 21:40:39,107 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-10 21:40:39,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:40:39,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:40:39,154 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:39,154 INFO HandlerThread:608881 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/tb" |
| save: true |
| } |
| control { |
| relay_id: "59e8947623744db792dc242bf1b20464" |
| } |
| _info { |
| stream_id: "s9hpsjf9" |
| } |
|
|
| 2024-12-10 21:40:39,154 DEBUG HandlerThread:608881 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-10 21:40:39,157 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/r8eyuers-code/FISH/eval_robot.py |
| 2024-12-10 21:40:39,170 INFO wandb-upload_0:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/aqm9dddc-wandb-metadata.json |
| 2024-12-10 21:40:39,186 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/h7dhr8h6-diff.patch |
| 2024-12-10 21:40:39,187 DEBUG SenderThread:608881 [sender.py:send():379] send: telemetry |
| 2024-12-10 21:40:39,204 DEBUG SenderThread:608881 [sender.py:send():379] send: tbrecord |
| 2024-12-10 21:40:39,205 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:40:39,206 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 with policy live |
| 2024-12-10 21:40:39,437 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/0eq0y07d-events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/code/FISH/eval_robot.py |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/diff.patch |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-metadata.json |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/requirements.txt |
| 2024-12-10 21:40:39,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:40:39,725 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/code/FISH |
| 2024-12-10 21:40:39,725 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/code |
| 2024-12-10 21:40:40,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:41,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:41,723 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:40:42,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:43,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:43,447 DEBUG SenderThread:608881 [sender.py:send():379] send: config |
| 2024-12-10 21:40:43,490 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:40:44,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:45,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:45,724 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:40:46,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:47,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:48,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:49,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:40:49,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:49,726 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:40:50,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:51,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:52,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:53,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:53,727 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:40:54,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:40:54,107 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:40:54,149 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:54,189 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:40:55,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:56,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:57,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:58,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:59,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:40:59,728 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:00,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:00,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:01,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:02,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:03,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:03,729 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:04,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:05,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:05,188 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:05,730 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:06,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:07,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:07,731 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:08,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:09,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:41:09,107 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:41:09,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:09,731 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:10,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:10,298 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:10,732 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/config.yaml |
| 2024-12-10 21:41:11,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:11,732 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:12,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:13,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:13,733 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:14,113 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:15,113 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:15,494 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:15,733 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:16,113 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:17,113 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:17,734 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:18,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:19,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:19,734 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:20,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:20,513 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:21,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:21,735 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:22,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:23,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:23,735 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:24,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:41:24,108 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:41:24,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:25,114 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:25,707 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:25,736 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:26,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:27,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:27,737 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:28,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:29,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:29,561 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:41:29,561 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 with policy now |
| 2024-12-10 21:41:29,561 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:41:29,562 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:41:29,562 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:41:29,563 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos |
| 2024-12-10 21:41:29,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media |
| 2024-12-10 21:41:29,827 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/35bhfxg3-media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 |
| 2024-12-10 21:41:30,108 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/oa60o3l4-media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 |
| 2024-12-10 21:41:30,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:31,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:31,567 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:31,738 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:32,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:33,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:33,739 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:34,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:35,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:36,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:37,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:37,122 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:37,740 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:38,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:38,795 DEBUG SystemMonitor:608881 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-10 21:41:38,796 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:41:39,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:41:39,107 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:41:39,149 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:40,116 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:41,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:41,742 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:41:42,003 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/mz351q7u-events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:41:42,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:42,229 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:43,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:43,743 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:44,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:45,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:46,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:47,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:47,234 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:47,744 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:48,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:49,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:49,745 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:50,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:51,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:51,745 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:52,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:52,431 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:53,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:53,746 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:54,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:41:54,108 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:41:54,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:55,119 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:55,747 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:56,119 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:57,119 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:57,451 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:41:57,748 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:41:58,119 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:59,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:41:59,748 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:00,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:01,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:01,749 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:02,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:02,646 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:03,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:03,750 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:04,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:05,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:05,750 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:06,120 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:07,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:07,751 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:07,837 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:08,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:08,796 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:42:09,107 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:42:09,108 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:42:09,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:09,752 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:10,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:11,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:11,752 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:12,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:13,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:13,257 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:13,753 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:13,856 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:42:13,856 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 with policy now |
| 2024-12-10 21:42:13,859 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:42:13,859 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:42:13,859 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:42:13,860 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:42:14,124 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:14,131 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/rsee72h5-media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 |
| 2024-12-10 21:42:14,753 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:42:14,754 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 |
| 2024-12-10 21:42:14,754 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval |
| 2024-12-10 21:42:15,088 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/4wjshnhu-media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 |
| 2024-12-10 21:42:15,124 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:15,754 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:16,126 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:16,754 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:17,126 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:17,754 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:18,125 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:18,926 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:19,127 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:20,127 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:21,127 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:21,756 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:22,127 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:23,127 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:24,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:42:24,108 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:42:24,150 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:24,263 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:25,128 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:26,128 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:27,128 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:28,128 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:29,128 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:30,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:30,129 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:31,129 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:32,129 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:33,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:34,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:35,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:35,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:36,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:37,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:38,132 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:38,796 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:42:39,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:42:39,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:42:39,154 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:40,131 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:40,246 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:41,130 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:42,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:43,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:44,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:45,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:46,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:46,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:47,134 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:48,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:48,612 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:42:48,612 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:42:48,614 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:42:48,764 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:42:48,764 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:42:49,018 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/dfwzg48s-events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:42:49,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:49,764 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:50,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:51,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:51,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:52,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:53,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:53,766 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:54,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:42:54,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:42:54,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:55,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:55,767 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:56,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:56,203 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:42:57,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:57,767 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:42:58,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:59,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:42:59,768 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:00,136 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:01,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:01,399 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:01,769 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:02,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:03,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:03,770 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:04,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:05,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:05,770 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:06,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:06,439 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:07,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:07,771 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:08,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:08,797 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:43:09,108 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:43:09,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:43:09,137 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:09,772 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:10,138 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:11,138 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:11,596 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:11,772 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:12,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:13,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:13,773 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:14,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:15,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:15,774 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:16,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:16,612 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:17,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:17,775 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:18,139 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:19,140 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:19,775 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:20,142 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:20,417 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:43:20,418 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:43:20,418 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_aac838f24394620216df.mp4 with policy now |
| 2024-12-10 21:43:20,419 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:43:20,419 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:43:20,419 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:43:20,700 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/srqd6xbw-media/videos/eval/2_eval_2_aac838f24394620216df.mp4 |
| 2024-12-10 21:43:20,776 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:43:20,776 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/2_eval_2_aac838f24394620216df.mp4 |
| 2024-12-10 21:43:20,776 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval |
| 2024-12-10 21:43:21,087 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/739rtuu0-media/videos/eval/2_eval_2_aac838f24394620216df.mp4 |
| 2024-12-10 21:43:21,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:21,776 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:22,143 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:22,435 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:22,776 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:23,144 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:24,117 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:43:24,117 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:43:24,144 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:25,143 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:25,777 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:26,143 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:27,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:27,778 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:28,018 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:28,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:29,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:29,779 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:43:30,144 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:30,444 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:43:30,444 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:43:30,445 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:43:30,779 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:43:31,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:31,779 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:32,153 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:33,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:33,402 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:34,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:35,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:35,781 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:36,145 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:37,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:37,781 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:38,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:38,516 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:38,797 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:43:39,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:43:39,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:43:39,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:39,782 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:40,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:41,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:41,783 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:42,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:43,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:43,723 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:43,784 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:44,181 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:45,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:45,785 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:46,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:47,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:47,785 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:48,146 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:48,919 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:49,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:49,786 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:50,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:51,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:51,787 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:52,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:53,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:53,787 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:53,921 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:54,109 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:43:54,109 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:43:54,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:55,147 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:55,788 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:56,148 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:57,148 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:57,789 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:43:58,148 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:59,115 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:43:59,157 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:43:59,790 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:00,148 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:01,151 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:01,790 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:02,150 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:02,301 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:44:02,302 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:44:02,302 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 with policy now |
| 2024-12-10 21:44:02,302 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:44:02,303 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:44:02,303 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:02,586 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/by1wmb7v-media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 |
| 2024-12-10 21:44:02,790 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:44:02,791 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 |
| 2024-12-10 21:44:02,791 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval |
| 2024-12-10 21:44:03,110 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/nincuu1n-media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 |
| 2024-12-10 21:44:03,155 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:03,791 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:04,118 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:04,154 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:04,791 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:05,181 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:05,792 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:06,160 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:07,170 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:08,158 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:08,798 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:44:09,111 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:44:09,111 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:44:09,161 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:09,264 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:09,793 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:10,162 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:10,489 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:44:10,489 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:44:10,490 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:10,793 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:44:10,793 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:44:11,073 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/mo50gl5v-events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:44:11,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:11,794 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:12,174 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:13,165 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:13,794 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:14,185 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:14,343 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:15,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:15,795 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:16,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:17,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:17,796 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:18,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:19,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:19,437 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:19,797 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:20,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:21,166 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:21,797 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:22,167 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:23,167 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:23,798 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:24,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:44:24,110 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:44:24,167 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:24,627 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:25,167 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:25,799 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:26,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:27,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:27,800 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:28,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:29,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:29,647 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:29,800 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:30,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:31,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:31,801 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:32,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:33,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:33,802 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:34,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:34,663 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:35,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:35,803 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:36,168 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:37,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:37,803 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:38,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:38,798 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:44:39,110 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:44:39,113 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:44:39,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:39,669 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:39,804 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:40,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:41,169 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:41,805 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:42,172 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:43,177 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:43,249 DEBUG SenderThread:608881 [sender.py:send():379] send: files |
| 2024-12-10 21:44:43,249 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 with policy now |
| 2024-12-10 21:44:43,251 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:44:43,252 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:44:43,252 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:44:43,252 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:43,556 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/09qsiei9-media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 |
| 2024-12-10 21:44:43,806 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:44:43,806 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:43,806 INFO Thread-12 :608881 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 |
| 2024-12-10 21:44:43,806 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval |
| 2024-12-10 21:44:44,132 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/e8z9uoxa-media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 |
| 2024-12-10 21:44:44,176 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:44,806 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:45,121 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:45,179 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:46,185 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:47,191 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:47,807 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:48,183 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:49,187 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:49,808 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:50,126 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:50,188 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:44:50,962 DEBUG SenderThread:608881 [sender.py:send():379] send: exit |
| 2024-12-10 21:44:50,962 INFO SenderThread:608881 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-10 21:44:50,962 INFO SenderThread:608881 [sender.py:send_exit():588] handling runtime: 252 |
| 2024-12-10 21:44:50,965 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:50,965 INFO SenderThread:608881 [sender.py:send_exit():594] send defer |
| 2024-12-10 21:44:50,966 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:50,966 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-10 21:44:50,966 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:50,966 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-10 21:44:50,966 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-10 21:44:50,966 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:50,966 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-10 21:44:50,966 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:50,966 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-10 21:44:50,966 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-10 21:44:50,966 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:50,966 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-10 21:44:50,966 INFO HandlerThread:608881 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-10 21:44:50,966 INFO HandlerThread:608881 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-10 21:44:50,967 DEBUG SystemMonitor:608881 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-10 21:44:50,967 INFO HandlerThread:608881 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-10 21:44:50,967 DEBUG SystemMonitor:608881 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-10 21:44:51,015 INFO HandlerThread:608881 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-10 21:44:51,015 INFO HandlerThread:608881 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-10 21:44:51,015 INFO HandlerThread:608881 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-10 21:44:51,016 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:51,016 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-10 21:44:51,016 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-10 21:44:51,016 DEBUG SenderThread:608881 [sender.py:send():379] send: stats |
| 2024-12-10 21:44:51,016 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:51,016 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-10 21:44:51,016 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:51,016 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-10 21:44:51,016 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-10 21:44:51,016 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:51,016 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-10 21:44:51,808 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:44:51,809 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:44:51,809 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:57,018 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:44:57,018 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:57,019 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:57,019 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-10 21:44:57,019 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-10 21:44:57,019 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-10 21:44:57,019 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:44:57,019 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:57,019 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:44:57,019 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-10 21:44:57,020 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:57,020 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:44:57,020 DEBUG SenderThread:608881 [sender.py:send():379] send: history |
| 2024-12-10 21:44:57,020 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:44:57,024 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:57,024 DEBUG SenderThread:608881 [sender.py:send():379] send: summary |
| 2024-12-10 21:44:57,024 INFO SenderThread:608881 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:44:57,024 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:57,024 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-10 21:44:57,024 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-10 21:44:57,024 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:57,024 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-10 21:44:57,025 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:57,025 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-10 21:44:57,025 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-10 21:44:57,025 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:44:57,025 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:57,025 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-10 21:44:57,025 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:57,025 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-10 21:44:57,810 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:44:57,963 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:44:57,990 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-10 21:44:57,990 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:44:57,990 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:57,991 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-10 21:44:57,991 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:57,991 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-10 21:44:57,991 INFO SenderThread:608881 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-10 21:44:57,993 INFO SenderThread:608881 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-10 21:44:58,007 INFO SenderThread:608881 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-10 21:44:58,022 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-10 21:44:58,023 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:58,023 DEBUG SenderThread:608881 [sender.py:send():379] send: artifact |
| 2024-12-10 21:44:58,023 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-10 21:44:58,597 INFO wandb-upload_0:608881 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp5dl4h77g |
| 2024-12-10 21:44:58,597 INFO wandb-upload_2:608881 [upload_job.py:push():85] Skipped uploading /tmp/tmpob1p2z0k/wandb-job.json |
| 2024-12-10 21:44:58,811 INFO Thread-12 :608881 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:44:58,861 INFO wandb-upload_1:608881 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpjyl59zur |
| 2024-12-10 21:44:58,963 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:44:59,526 INFO SenderThread:608881 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM2NjgxMjE3Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM2NjgwNjQxOA==', 'versionIndex': 343}}} |
| 2024-12-10 21:44:59,526 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:59,526 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-10 21:44:59,526 INFO SenderThread:608881 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-10 21:44:59,811 INFO SenderThread:608881 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files |
| 2024-12-10 21:44:59,812 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/config.yaml config.yaml |
| 2024-12-10 21:44:59,812 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json wandb-summary.json |
| 2024-12-10 21:44:59,812 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/requirements.txt requirements.txt |
| 2024-12-10 21:44:59,813 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-10 21:44:59,815 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:44:59,816 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log output.log |
| 2024-12-10 21:44:59,817 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/diff.patch diff.patch |
| 2024-12-10 21:44:59,817 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-10 21:44:59,817 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 media/videos/eval/1_eval_1_0f1a83ed932af73a96be.mp4 |
| 2024-12-10 21:44:59,817 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 media/videos/eval/3_eval_3_9f3ddf93c395db4f718a.mp4 |
| 2024-12-10 21:44:59,818 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 media/videos/eval/0_eval_0_99cde89795fdf8f2318a.mp4 |
| 2024-12-10 21:44:59,820 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 media/videos/eval/4_eval_4_86ec182da7a62ff7b578.mp4 |
| 2024-12-10 21:44:59,820 INFO SenderThread:608881 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/media/videos/eval/2_eval_2_aac838f24394620216df.mp4 media/videos/eval/2_eval_2_aac838f24394620216df.mp4 |
| 2024-12-10 21:44:59,820 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-10 21:44:59,820 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:44:59,820 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:44:59,820 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-10 21:44:59,820 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:44:59,820 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-10 21:44:59,820 INFO SenderThread:608881 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-10 21:44:59,964 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:44:59,964 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:45:00,000 INFO wandb-upload_2:608881 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/requirements.txt |
| 2024-12-10 21:45:00,030 INFO wandb-upload_1:608881 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/wandb-summary.json |
| 2024-12-10 21:45:00,058 INFO wandb-upload_0:608881 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/config.yaml |
| 2024-12-10 21:45:00,095 INFO wandb-upload_4:608881 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/files/output.log |
| 2024-12-10 21:45:00,103 INFO wandb-upload_3:608881 [upload_job.py:push():130] Uploaded file /tmp/tmp0jm5xe66wandb/j6opz80u-events.out.tfevents.1733884839.leonmkim-ROG-Strix-G15CS-G15CS.608694.0 |
| 2024-12-10 21:45:00,303 INFO Thread-11 (_thread_body):608881 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-10 21:45:00,303 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:45:00,303 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-10 21:45:00,304 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:45:00,304 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-10 21:45:00,304 INFO SenderThread:608881 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-10 21:45:00,305 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-10 21:45:00,305 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:45:00,305 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-10 21:45:00,305 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:45:00,305 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-10 21:45:00,305 INFO SenderThread:608881 [file_stream.py:finish():601] file stream finish called |
| 2024-12-10 21:45:00,501 INFO SenderThread:608881 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-10 21:45:00,502 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-10 21:45:00,502 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:45:00,502 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-10 21:45:00,502 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:45:00,502 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-10 21:45:00,502 INFO SenderThread:608881 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-10 21:45:00,502 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:45:00,502 INFO HandlerThread:608881 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-10 21:45:00,502 DEBUG SenderThread:608881 [sender.py:send():379] send: final |
| 2024-12-10 21:45:00,503 DEBUG SenderThread:608881 [sender.py:send():379] send: footer |
| 2024-12-10 21:45:00,503 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:45:00,503 INFO SenderThread:608881 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-10 21:45:00,503 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:45:00,503 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:45:00,503 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-10 21:45:00,503 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: server_info |
| 2024-12-10 21:45:00,505 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:45:00,505 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-10 21:45:00,505 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-10 21:45:00,506 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:45:00,560 DEBUG SenderThread:608881 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:45:00,560 INFO MainThread:608881 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-10 21:45:00,560 INFO MainThread:608881 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-10 21:45:00,560 INFO MainThread:608881 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-10 21:45:00,560 DEBUG HandlerThread:608881 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-10 21:45:00,560 INFO HandlerThread:608881 [handler.py:finish():882] shutting down handler |
| 2024-12-10 21:45:01,505 INFO WriterThread:608881 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/214032/wandb/run-20241210_214038-s9hpsjf9/run-s9hpsjf9.wandb |
| 2024-12-10 21:45:01,560 INFO SenderThread:608881 [sender.py:finish():1615] shutting down sender |
| 2024-12-10 21:45:01,560 INFO SenderThread:608881 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-10 21:45:01,560 INFO SenderThread:608881 [file_pusher.py:join():175] waiting for file pusher |
|
|
