| 2025-01-06 17:49:44,899 INFO StreamThr :2094664 [internal.py:wandb_internal():85] W&B internal server running at pid: 2094664, started at: 2025-01-06 17:49:44.898786 |
| 2025-01-06 17:49:44,900 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 17:49:44,911 INFO WriterThread:2094664 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/run-h2bqbukm.wandb |
| 2025-01-06 17:49:44,912 DEBUG SenderThread:2094664 [sender.py:send():379] send: header |
| 2025-01-06 17:49:44,920 DEBUG SenderThread:2094664 [sender.py:send():379] send: run |
| 2025-01-06 17:49:45,159 INFO SenderThread:2094664 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files |
| 2025-01-06 17:49:45,159 INFO SenderThread:2094664 [sender.py:_start_run_threads():1188] run started: h2bqbukm with start time 1736203784.899512 |
| 2025-01-06 17:49:45,174 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 17:49:45,174 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 17:49:45,247 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 17:49:45,252 DEBUG HandlerThread:2094664 [system_info.py:__init__():26] System info init |
| 2025-01-06 17:49:45,252 DEBUG HandlerThread:2094664 [system_info.py:__init__():41] System info init done |
| 2025-01-06 17:49:45,252 INFO HandlerThread:2094664 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 17:49:45,252 INFO SystemMonitor:2094664 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 17:49:45,253 INFO HandlerThread:2094664 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 17:49:45,253 INFO SystemMonitor:2094664 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 17:49:45,254 INFO SystemMonitor:2094664 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 17:49:45,254 INFO SystemMonitor:2094664 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 17:49:45,254 INFO SystemMonitor:2094664 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 17:49:45,255 INFO SystemMonitor:2094664 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 17:49:45,265 DEBUG HandlerThread:2094664 [system_info.py:probe():152] Probing system |
| 2025-01-06 17:49:45,267 DEBUG HandlerThread:2094664 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 17:49:45,271 DEBUG HandlerThread:2094664 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 17:49:45,272 DEBUG HandlerThread:2094664 [system_info.py:probe():200] Probing system done |
| 2025-01-06 17:49:45,272 DEBUG HandlerThread:2094664 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-06T22:49:45.265888', 'startedAt': '2025-01-06T22:49:44.891249', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3748.318, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.021, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.905, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 365.2649574279785}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 17:49:45,272 INFO HandlerThread:2094664 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 17:49:45,272 INFO HandlerThread:2094664 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 17:49:45,272 DEBUG HandlerThread:2094664 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 17:49:45,273 DEBUG HandlerThread:2094664 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 17:49:45,273 DEBUG HandlerThread:2094664 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 17:49:45,342 DEBUG HandlerThread:2094664 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 17:49:45,342 INFO HandlerThread:2094664 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 17:49:45,346 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:49:45,346 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 17:49:45,346 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 17:49:45,346 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 17:49:45,577 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/cuib13ev-wandb-metadata.json |
| 2025-01-06 17:49:45,590 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 17:49:45,591 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 17:49:45,591 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:49:45,591 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:49:45,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:45,593 INFO HandlerThread:2094664 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/tb" |
| save: true |
| } |
| control { |
| relay_id: "358343e3048243c9bc65db54a5239cdf" |
| } |
| _info { |
| stream_id: "h2bqbukm" |
| } |
|
|
| 2025-01-06 17:49:45,593 DEBUG HandlerThread:2094664 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 17:49:45,610 INFO wandb-upload_2:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/02g9cnb6-diff.patch |
| 2025-01-06 17:49:45,613 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/ngzqhp3e-code/FISH/eval_robot.py |
| 2025-01-06 17:49:45,703 DEBUG SenderThread:2094664 [sender.py:send():379] send: telemetry |
| 2025-01-06 17:49:45,703 DEBUG SenderThread:2094664 [sender.py:send():379] send: tbrecord |
| 2025-01-06 17:49:45,704 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:49:45,704 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 with policy live |
| 2025-01-06 17:49:45,878 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/4ughl6zr-events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/requirements.txt |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/diff.patch |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/code/FISH/eval_robot.py |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-metadata.json |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/code |
| 2025-01-06 17:49:46,161 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/code/FISH |
| 2025-01-06 17:49:46,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:47,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:48,160 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:49:48,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:49,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:50,591 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:49:50,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:51,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:52,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:53,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:54,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:55,591 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:49:55,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:55,698 DEBUG SenderThread:2094664 [sender.py:send():379] send: config |
| 2025-01-06 17:49:56,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:57,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:58,163 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:49:58,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:49:59,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:00,591 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:50:00,591 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:50:00,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:00,645 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:01,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:02,164 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:02,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:03,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:04,164 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:04,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:05,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:06,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:06,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:07,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:08,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:09,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:10,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:11,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:11,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:12,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:13,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:14,167 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:14,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:15,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:50:15,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:50:15,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:16,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:17,153 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:17,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:18,168 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:18,168 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/config.yaml |
| 2025-01-06 17:50:18,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:19,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:20,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:21,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:22,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:22,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:23,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:24,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:25,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:26,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:27,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:28,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:28,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:29,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:30,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:50:30,594 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:50:30,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:31,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:32,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:33,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:33,680 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:34,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:35,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:36,173 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:36,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:37,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:38,173 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:38,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:39,157 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:39,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:40,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:41,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:42,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:43,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:44,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:44,600 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:45,255 DEBUG SystemMonitor:2094664 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 17:50:45,256 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:50:45,591 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:50:45,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:50:45,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:46,601 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:47,601 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:48,601 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:49,602 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:49,697 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:50,602 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:51,603 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:51,681 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:50:51,682 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:50:51,682 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 with policy now |
| 2025-01-06 17:50:51,682 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:50:51,682 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:50:51,684 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:50:51,905 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/fkqxc7ko-media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 |
| 2025-01-06 17:50:52,177 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:52,177 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:50:52,177 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 |
| 2025-01-06 17:50:52,178 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval |
| 2025-01-06 17:50:52,178 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos |
| 2025-01-06 17:50:52,178 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media |
| 2025-01-06 17:50:52,458 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/u2peld71-media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 |
| 2025-01-06 17:50:52,603 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:53,603 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:54,178 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:54,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:55,264 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:50:55,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:56,179 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:56,603 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:57,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:58,179 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:50:58,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:50:59,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:00,180 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:00,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:51:00,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:51:00,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:00,663 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:01,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:02,604 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:03,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:04,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:05,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:06,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:06,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:07,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:08,605 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:09,183 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:51:09,423 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/4r935k1e-events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:51:09,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:10,183 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:10,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:11,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:12,184 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:12,473 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:12,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:13,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:14,185 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:14,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:15,257 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:51:15,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:51:15,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:51:15,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:16,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:17,606 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:17,666 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:18,608 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:19,607 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:20,608 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:21,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:22,608 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:23,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:23,608 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:24,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:25,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:26,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:27,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:28,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:28,609 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:29,610 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:30,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:51:30,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:51:30,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:31,610 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:32,610 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:33,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:33,694 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:34,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:35,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:36,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:37,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:38,611 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:39,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:39,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:40,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:41,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:42,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:43,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:44,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:44,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:45,260 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:51:45,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:51:45,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:51:45,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:46,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:47,612 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:48,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:48,818 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:51:48,818 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 with policy now |
| 2025-01-06 17:51:48,819 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:51:48,819 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:51:48,819 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:51:48,834 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:51:49,031 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/i5xminzo-media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 |
| 2025-01-06 17:51:49,196 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:51:49,196 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 |
| 2025-01-06 17:51:49,196 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval |
| 2025-01-06 17:51:49,456 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/iemwg3av-media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 |
| 2025-01-06 17:51:49,620 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:49,852 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:50,196 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:50,618 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:51,196 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:51,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:52,620 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:53,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:54,197 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:54,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:55,245 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:51:55,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:56,198 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:51:56,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:57,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:58,620 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:51:59,619 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:00,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:52:00,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:52:00,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:00,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:01,620 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:02,620 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:03,645 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:04,027 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:52:04,027 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:52:04,028 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:52:04,201 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:52:04,201 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:52:04,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:05,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:06,201 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:06,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:06,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:07,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:08,202 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:08,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:09,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:10,203 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:10,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:11,621 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:12,597 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:12,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:13,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:14,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:15,257 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:52:15,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:52:15,592 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:52:15,633 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:16,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:17,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:17,664 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:18,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:19,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:20,622 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:21,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:22,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:23,410 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:23,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:24,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:25,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:26,208 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:26,623 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:27,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:28,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:28,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:29,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:30,592 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:52:30,593 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:52:30,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:31,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:32,624 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:33,625 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:33,691 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:34,665 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:35,625 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:36,212 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:36,625 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:37,625 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:38,625 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:39,598 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:39,626 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:40,626 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:41,626 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:42,626 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:43,630 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:43,697 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:52:43,697 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 with policy now |
| 2025-01-06 17:52:43,702 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:52:43,703 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:52:43,703 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:52:43,741 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:52:43,943 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/45ske50i-media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 |
| 2025-01-06 17:52:44,214 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:44,215 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:52:44,215 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 |
| 2025-01-06 17:52:44,215 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval |
| 2025-01-06 17:52:44,493 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/gcu1z4gq-media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 |
| 2025-01-06 17:52:44,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:44,742 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:45,215 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:45,258 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:52:45,602 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:52:45,602 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:52:45,645 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:46,215 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:46,643 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:47,645 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:48,216 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:48,647 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:49,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:50,026 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:50,645 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:51,644 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:51,864 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:52:51,865 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:52:51,865 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:52:52,217 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:52:52,217 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:52,217 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:52:52,471 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/daonpq4k-events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:52:52,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:53,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:54,218 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:54,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:55,284 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:52:55,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:56,219 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:56,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:57,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:58,219 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:52:58,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:52:59,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:00,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:53:00,594 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:53:00,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:00,680 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:01,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:02,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:03,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:04,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:05,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:06,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:06,649 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:07,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:08,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:09,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:10,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:11,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:12,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:12,650 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:13,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:14,225 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:14,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:15,258 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:53:15,593 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:53:15,594 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:53:15,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:16,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:17,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:17,690 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:18,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:19,651 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:20,652 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:21,652 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:22,652 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:23,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:23,652 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:24,652 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:25,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:26,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:27,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:28,599 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:28,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:29,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:30,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:53:30,594 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:53:30,653 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:31,657 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:31,851 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:53:31,851 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 with policy now |
| 2025-01-06 17:53:31,856 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:53:31,856 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:53:31,857 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:53:31,857 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:53:32,079 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/oovtj40c-media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 |
| 2025-01-06 17:53:32,231 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:53:32,232 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:32,232 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 |
| 2025-01-06 17:53:32,232 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval |
| 2025-01-06 17:53:32,490 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/pd97ru4o-media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 |
| 2025-01-06 17:53:32,669 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:33,232 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:33,663 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:33,901 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:34,233 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:34,671 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:35,669 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:36,666 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:37,666 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:38,234 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:38,662 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:39,345 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:39,668 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:40,235 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:40,666 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:41,667 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:41,928 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:53:41,929 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:53:41,929 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:53:42,236 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:53:42,236 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:42,236 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:53:42,671 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:43,670 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:44,237 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:44,517 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:44,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:45,259 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:53:45,595 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:53:45,596 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:53:45,701 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:46,238 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:46,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:47,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:48,238 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:53:48,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:49,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:49,702 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:50,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:51,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:52,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:53,673 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:54,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:55,601 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:53:55,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:56,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:57,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:58,674 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:53:59,675 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:00,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:54:00,595 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:54:00,675 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:00,736 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:01,675 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:02,675 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:03,676 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:04,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:05,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:06,601 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:06,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:07,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:08,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:09,677 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:10,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:11,602 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:11,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:12,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:13,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:14,679 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:15,259 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:54:15,594 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:54:15,595 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:54:15,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:16,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:16,743 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:17,678 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:18,679 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:19,679 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:20,680 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:21,634 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:54:21,634 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 with policy now |
| 2025-01-06 17:54:21,637 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:54:21,637 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:54:21,637 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:21,638 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:21,684 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:21,854 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/urkczbbw-media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 |
| 2025-01-06 17:54:22,250 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:22,250 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:22,250 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 |
| 2025-01-06 17:54:22,250 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval |
| 2025-01-06 17:54:22,513 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/k7v97fhr-media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 |
| 2025-01-06 17:54:22,669 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:22,694 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:23,691 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:24,251 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:24,690 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:25,691 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:26,251 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:26,695 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:27,694 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:28,130 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:28,695 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:29,697 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:30,253 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:30,596 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:54:30,596 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:54:30,667 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 17:54:30,668 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 17:54:30,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:30,847 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:54:31,211 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 17:54:31,222 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:54:31,253 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:54:31,253 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json |
| 2025-01-06 17:54:31,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 17:54:31,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media |
| 2025-01-06 17:54:31,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table/eval |
| 2025-01-06 17:54:31,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table |
| 2025-01-06 17:54:31,319 INFO wandb-upload_1:2094664 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpvwfv0v40 |
| 2025-01-06 17:54:31,472 INFO wandb-upload_2:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/5qcbqjs6-media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json |
| 2025-01-06 17:54:31,473 INFO wandb-upload_3:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/b1i4xoxx-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 17:54:31,572 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/yxglis5v-events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:54:31,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:31,702 INFO SenderThread:2094664 [sender.py:send_request_log_artifact():1525] logged artifact run-h2bqbukm-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTk2NzA4Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxNTgxOA==', 'latestArtifact': None}} |
| 2025-01-06 17:54:31,702 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:54:31,702 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-06 17:54:31,703 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:54:31,703 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,703 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,703 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:54:31,704 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,704 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,704 DEBUG SenderThread:2094664 [sender.py:send():379] send: telemetry |
| 2025-01-06 17:54:31,704 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,704 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,705 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,705 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,705 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,705 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,707 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,707 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,707 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,708 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,708 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,708 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,708 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,709 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,709 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,709 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,709 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,710 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,710 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,710 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,710 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:31,711 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:31,711 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 17:54:31,936 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/5079k4fk-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 17:54:32,120 INFO wandb-upload_3:2094664 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphij2ei1v |
| 2025-01-06 17:54:32,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:32,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:32,562 INFO SenderThread:2094664 [sender.py:send_request_log_artifact():1525] logged artifact run-h2bqbukm-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTk2NzExMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxNTgyMw==', 'latestArtifact': None}} |
| 2025-01-06 17:54:32,562 DEBUG SenderThread:2094664 [sender.py:send():379] send: files |
| 2025-01-06 17:54:32,562 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json with policy now |
| 2025-01-06 17:54:32,562 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:54:32,562 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:32,563 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:32,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:32,815 INFO wandb-upload_2:2094664 [upload_job.py:push():130] Uploaded file /tmp/tmpajowzz_4wandb/lbt94spa-media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json |
| 2025-01-06 17:54:33,254 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:33,563 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:33,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:34,255 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:34,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:35,693 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:36,633 DEBUG SenderThread:2094664 [sender.py:send():379] send: exit |
| 2025-01-06 17:54:36,633 INFO SenderThread:2094664 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 17:54:36,634 INFO SenderThread:2094664 [sender.py:send_exit():588] handling runtime: 291 |
| 2025-01-06 17:54:36,635 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:36,635 INFO SenderThread:2094664 [sender.py:send_exit():594] send defer |
| 2025-01-06 17:54:36,635 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:36,635 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 17:54:36,635 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:36,635 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 17:54:36,635 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 17:54:36,635 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:36,635 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 17:54:36,636 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:36,636 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 17:54:36,636 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 17:54:36,636 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:36,636 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 17:54:36,636 INFO HandlerThread:2094664 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 17:54:36,636 DEBUG SystemMonitor:2094664 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 17:54:36,636 DEBUG SystemMonitor:2094664 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 17:54:36,637 INFO HandlerThread:2094664 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 17:54:36,637 INFO HandlerThread:2094664 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 17:54:36,688 INFO HandlerThread:2094664 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 17:54:36,688 INFO HandlerThread:2094664 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 17:54:36,688 INFO HandlerThread:2094664 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 17:54:36,689 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:36,689 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 17:54:36,689 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 17:54:36,689 DEBUG SenderThread:2094664 [sender.py:send():379] send: stats |
| 2025-01-06 17:54:36,689 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:36,689 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 17:54:36,689 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:36,690 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 17:54:36,690 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 17:54:36,690 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:36,690 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 17:54:37,256 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:38,256 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:39,257 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/config.yaml |
| 2025-01-06 17:54:43,009 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:54:43,009 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:43,009 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 17:54:43,011 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:43,011 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 17:54:43,011 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 17:54:43,011 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:54:43,011 DEBUG SenderThread:2094664 [sender.py:send():379] send: history |
| 2025-01-06 17:54:43,011 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:54:43,011 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:43,011 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 17:54:43,012 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:43,012 DEBUG SenderThread:2094664 [sender.py:send():379] send: summary |
| 2025-01-06 17:54:43,016 INFO SenderThread:2094664 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:54:43,016 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:43,016 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 17:54:43,016 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 17:54:43,016 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:43,016 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 17:54:43,016 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:43,016 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 17:54:43,017 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 17:54:43,017 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:54:43,017 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:43,017 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 17:54:43,017 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:43,017 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 17:54:43,258 INFO Thread-12 :2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:43,635 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:54:44,905 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 17:54:44,905 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:54:44,906 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:44,906 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 17:54:44,906 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:44,906 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 17:54:44,906 INFO SenderThread:2094664 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 17:54:44,908 INFO SenderThread:2094664 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 17:54:44,910 INFO SenderThread:2094664 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 17:54:44,926 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 17:54:44,927 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:44,927 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 17:54:44,927 DEBUG SenderThread:2094664 [sender.py:send():379] send: artifact |
| 2025-01-06 17:54:45,154 INFO SenderThread:2094664 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTk1OTYxMQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk1OTYxMQ==', 'versionIndex': 425}}} |
| 2025-01-06 17:54:45,154 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,154 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 17:54:45,154 INFO SenderThread:2094664 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 17:54:45,259 INFO SenderThread:2094664 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/config.yaml config.yaml |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/requirements.txt requirements.txt |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 events.out.tfevents.1736203785.leonmkim-ROG-Strix-G15CS-G15CS.2094491.0 |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log output.log |
| 2025-01-06 17:54:45,260 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/diff.patch diff.patch |
| 2025-01-06 17:54:45,266 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 17:54:45,266 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 media/videos/eval/0_eval_0_16e69bedd16e7b6c3536.mp4 |
| 2025-01-06 17:54:45,266 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 media/videos/eval/3_eval_3_66a85b9e84c70fcc60b8.mp4 |
| 2025-01-06 17:54:45,266 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 media/videos/eval/4_eval_4_ee426026d86a22c1b5e5.mp4 |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 media/videos/eval/2_eval_2_90c1047e360390350d75.mp4 |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 media/videos/eval/1_eval_1_57b085b1bc4aeda7344c.mp4 |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json media/table/eval/total_success_rate_ci_6_1e52f8fa0b3eaff8361f.table.json |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 17:54:45,267 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:45,267 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 17:54:45,267 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 17:54:45,267 INFO SenderThread:2094664 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 17:54:45,428 INFO wandb-upload_3:2094664 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/wandb-summary.json |
| 2025-01-06 17:54:45,435 INFO wandb-upload_0:2094664 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/requirements.txt |
| 2025-01-06 17:54:45,449 INFO wandb-upload_1:2094664 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/config.yaml |
| 2025-01-06 17:54:45,484 INFO wandb-upload_2:2094664 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/files/output.log |
| 2025-01-06 17:54:45,636 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:54:45,636 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:54:45,684 INFO Thread-11 (_thread_body):2094664 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 17:54:45,684 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:45,684 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 17:54:45,684 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,685 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 17:54:45,685 INFO SenderThread:2094664 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 17:54:45,686 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 17:54:45,686 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:45,686 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 17:54:45,686 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,686 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 17:54:45,686 INFO SenderThread:2094664 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 17:54:45,884 INFO SenderThread:2094664 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 17:54:45,884 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 17:54:45,884 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:45,885 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 17:54:45,885 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,885 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 17:54:45,885 INFO SenderThread:2094664 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 17:54:45,885 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:54:45,885 INFO HandlerThread:2094664 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 17:54:45,886 DEBUG SenderThread:2094664 [sender.py:send():379] send: final |
| 2025-01-06 17:54:45,886 DEBUG SenderThread:2094664 [sender.py:send():379] send: footer |
| 2025-01-06 17:54:45,886 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:54:45,886 INFO SenderThread:2094664 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 17:54:45,886 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:54:45,887 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 17:54:45,887 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:54:45,887 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 17:54:45,887 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 17:54:45,887 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:54:45,889 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 17:54:45,892 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:54:45,943 DEBUG SenderThread:2094664 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:54:45,943 INFO MainThread:2094664 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 17:54:45,944 INFO MainThread:2094664 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 17:54:45,944 INFO MainThread:2094664 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 17:54:45,944 DEBUG HandlerThread:2094664 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 17:54:45,944 INFO HandlerThread:2094664 [handler.py:finish():882] shutting down handler |
| 2025-01-06 17:54:46,887 INFO WriterThread:2094664 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1003_0/h2bqbukm/wandb/run-20250106_174944-h2bqbukm/run-h2bqbukm.wandb |
| 2025-01-06 17:54:46,943 INFO SenderThread:2094664 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 17:54:46,943 INFO SenderThread:2094664 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 17:54:46,943 INFO SenderThread:2094664 [file_pusher.py:join():175] waiting for file pusher |
|
|
