| 2025-01-17 16:46:09,623 INFO StreamThr :2525727 [internal.py:wandb_internal():85] W&B internal server running at pid: 2525727, started at: 2025-01-17 16:46:09.623178 |
| 2025-01-17 16:46:09,624 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status |
| 2025-01-17 16:46:09,637 INFO WriterThread:2525727 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/run-ekswld3y.wandb |
| 2025-01-17 16:46:09,637 DEBUG SenderThread:2525727 [sender.py:send():379] send: header |
| 2025-01-17 16:46:09,648 DEBUG SenderThread:2525727 [sender.py:send():379] send: run |
| 2025-01-17 16:46:09,905 INFO SenderThread:2525727 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files |
| 2025-01-17 16:46:09,905 INFO SenderThread:2525727 [sender.py:_start_run_threads():1188] run started: ekswld3y with start time 1737150369.623394 |
| 2025-01-17 16:46:09,920 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-17 16:46:09,920 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: check_version |
| 2025-01-17 16:46:09,992 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-17 16:46:09,997 DEBUG HandlerThread:2525727 [system_info.py:__init__():26] System info init |
| 2025-01-17 16:46:09,997 DEBUG HandlerThread:2525727 [system_info.py:__init__():41] System info init done |
| 2025-01-17 16:46:09,997 INFO HandlerThread:2525727 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-17 16:46:09,997 INFO SystemMonitor:2525727 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-17 16:46:09,998 INFO HandlerThread:2525727 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-17 16:46:09,999 INFO SystemMonitor:2525727 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-17 16:46:09,999 INFO SystemMonitor:2525727 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-17 16:46:10,000 INFO SystemMonitor:2525727 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-17 16:46:10,000 INFO SystemMonitor:2525727 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-17 16:46:10,001 INFO SystemMonitor:2525727 [interfaces.py:start():188] Started network monitoring |
| 2025-01-17 16:46:10,011 DEBUG HandlerThread:2525727 [system_info.py:probe():152] Probing system |
| 2025-01-17 16:46:10,012 DEBUG HandlerThread:2525727 [system_info.py:_probe_git():137] Probing git |
| 2025-01-17 16:46:10,017 DEBUG HandlerThread:2525727 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-17 16:46:10,017 DEBUG HandlerThread:2525727 [system_info.py:probe():200] Probing system done |
| 2025-01-17 16:46:10,018 DEBUG HandlerThread:2525727 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-17T21:46:10.011408', 'startedAt': '2025-01-17T21:46:09.609936', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3570.758, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4525.43, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4514.673, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 442.17910385131836}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-17 16:46:10,018 INFO HandlerThread:2525727 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-17 16:46:10,018 INFO HandlerThread:2525727 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-17 16:46:10,022 DEBUG HandlerThread:2525727 [system_info.py:_save_code():44] Saving code |
| 2025-01-17 16:46:10,024 DEBUG HandlerThread:2525727 [system_info.py:_save_code():67] Saving code done |
| 2025-01-17 16:46:10,024 DEBUG HandlerThread:2525727 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-17 16:46:10,118 DEBUG HandlerThread:2525727 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-17 16:46:10,119 INFO HandlerThread:2525727 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-17 16:46:10,122 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:46:10,122 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-17 16:46:10,122 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-17 16:46:10,123 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-17 16:46:10,123 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-17 16:46:10,363 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-17 16:46:10,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-17 16:46:10,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:46:10,367 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:46:10,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:10,370 INFO HandlerThread:2525727 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/tb" |
| save: true |
| } |
| control { |
| relay_id: "8cd42fab814e458a8ab6950d71c030ed" |
| } |
| _info { |
| stream_id: "ekswld3y" |
| } |
|
|
| 2025-01-17 16:46:10,370 DEBUG HandlerThread:2525727 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-17 16:46:10,382 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/752jmyh8-code/FISH/eval_robot.py |
| 2025-01-17 16:46:10,383 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/s2qqj5rd-wandb-metadata.json |
| 2025-01-17 16:46:10,388 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/dcimhife-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 16:46:10,413 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/2dvfwrrg-diff.patch |
| 2025-01-17 16:46:10,453 DEBUG SenderThread:2525727 [sender.py:send():379] send: telemetry |
| 2025-01-17 16:46:10,453 DEBUG SenderThread:2525727 [sender.py:send():379] send: tbrecord |
| 2025-01-17 16:46:10,454 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:46:10,454 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 with policy live |
| 2025-01-17 16:46:10,646 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/eehqmqzn-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:46:10,906 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/code/FISH/eval_robot.py |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-metadata.json |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/diff.patch |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/requirements.txt |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/code |
| 2025-01-17 16:46:10,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/code/FISH |
| 2025-01-17 16:46:11,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:12,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:12,906 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:13,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:14,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:15,223 DEBUG SenderThread:2525727 [sender.py:send():379] send: config |
| 2025-01-17 16:46:15,223 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:15,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:16,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:16,907 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:17,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:18,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:18,908 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:19,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:20,242 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:20,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:21,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:22,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:22,909 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:23,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:24,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:24,909 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:25,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:46:25,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:46:25,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:25,460 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:26,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:26,910 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:46:27,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:28,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:29,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:30,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:31,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:31,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:32,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:33,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:34,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:35,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:36,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:36,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:37,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:38,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:39,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:40,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:46:40,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:46:40,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:41,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:41,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:41,914 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/config.yaml |
| 2025-01-17 16:46:42,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:43,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:44,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:45,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:46,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:47,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:47,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:48,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:49,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:50,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:51,370 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:52,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:52,370 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:53,370 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:54,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:55,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:46:55,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:46:55,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:56,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:57,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:57,490 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:46:58,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:46:59,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:00,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:00,919 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:01,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:02,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:02,830 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:02,920 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:03,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:04,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:04,921 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:05,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:06,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:07,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:08,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:08,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:09,372 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:10,001 DEBUG SystemMonitor:2525727 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-17 16:47:10,002 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:47:10,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:47:10,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:47:10,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:11,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:12,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:13,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:13,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:14,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:15,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:16,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:17,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:18,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:19,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:19,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:20,373 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:21,374 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:22,374 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:23,374 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:24,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:24,374 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:25,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:47:25,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:47:25,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:26,374 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:27,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:28,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:29,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:29,432 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:30,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:30,786 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:47:30,786 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 with policy now |
| 2025-01-17 16:47:30,786 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:47:30,787 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:47:30,787 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:47:30,788 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:47:30,928 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:30,928 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:47:30,928 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 |
| 2025-01-17 16:47:30,928 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media |
| 2025-01-17 16:47:30,929 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos |
| 2025-01-17 16:47:30,929 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:47:31,008 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/lkxfyqn7-media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 |
| 2025-01-17 16:47:31,277 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/wlrx3qce-media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 |
| 2025-01-17 16:47:31,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:32,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:32,929 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:33,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:34,376 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:34,445 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:35,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:36,376 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:36,930 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:37,376 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:38,376 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:38,931 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:39,376 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:40,002 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:47:40,002 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:40,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:47:40,364 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:47:40,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:41,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:42,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:43,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:44,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:45,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:45,377 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:46,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:47,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:47,934 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:47:48,188 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/y97lkd40-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:47:48,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:48,934 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:49,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:50,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:50,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:50,935 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:51,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:52,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:52,936 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:47:53,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:54,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:55,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:47:55,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:47:55,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:55,480 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:47:56,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:57,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:58,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:47:59,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:00,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:01,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:01,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:02,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:03,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:04,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:05,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:06,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:06,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:07,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:08,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:09,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:10,008 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:48:10,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:48:10,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:48:10,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:11,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:11,428 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:12,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:13,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:14,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:15,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:16,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:17,229 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:17,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:18,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:18,944 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:19,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:19,389 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:48:19,389 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 with policy now |
| 2025-01-17 16:48:19,429 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:48:19,430 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:48:19,430 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:48:19,430 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:48:19,614 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/uj5remwf-media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 |
| 2025-01-17 16:48:19,944 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:48:19,945 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 |
| 2025-01-17 16:48:19,945 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:48:20,246 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/qj5nohh8-media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 |
| 2025-01-17 16:48:20,387 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:20,945 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:21,385 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:22,370 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:22,387 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:23,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:24,392 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:24,946 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:25,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:48:25,367 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:48:25,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:26,385 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:26,947 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:27,385 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:27,833 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:28,385 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:29,386 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:29,948 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:48:30,387 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:30,838 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:48:30,839 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:48:30,839 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:48:30,948 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:30,948 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:48:31,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:32,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:32,937 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:32,949 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:33,387 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:34,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:34,951 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:35,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:36,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:36,952 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:37,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:38,174 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:38,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:39,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:40,003 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:48:40,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:48:40,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:48:40,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:40,954 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:48:41,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:42,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:43,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:43,440 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:44,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:45,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:46,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:47,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:48,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:49,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:49,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:50,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:51,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:52,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:53,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:54,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:48:54,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:55,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:48:55,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:48:55,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:56,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:57,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:58,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:59,391 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:48:59,459 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:00,390 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:01,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:01,589 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:49:01,590 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:49:01,590 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 with policy now |
| 2025-01-17 16:49:01,590 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:49:01,590 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:49:01,591 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:49:01,811 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/nx6k8p3v-media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 |
| 2025-01-17 16:49:01,961 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:49:01,961 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 |
| 2025-01-17 16:49:01,962 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:49:02,254 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/wegpzwhv-media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 |
| 2025-01-17 16:49:02,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:02,962 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:03,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:04,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:04,607 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:05,408 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:06,403 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:06,963 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:07,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:08,402 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:08,964 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:09,403 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:09,643 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:49:09,643 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:49:09,643 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:09,644 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:49:09,965 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:49:09,965 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:49:10,003 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:49:10,220 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/pi4kdv8o-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:49:10,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:49:10,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:49:10,408 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:10,965 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:11,411 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:12,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:12,966 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:13,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:14,403 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:14,779 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:14,966 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:15,403 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:16,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:17,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:18,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:18,968 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:19,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:20,357 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:20,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:20,969 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:21,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:22,404 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:23,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:24,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:25,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:49:25,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:49:25,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:25,505 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:26,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:27,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:28,405 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:29,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:30,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:30,973 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:31,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:31,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:32,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:33,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:34,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:35,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:36,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:36,761 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:36,975 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:37,407 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:38,407 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:39,407 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:40,004 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:49:40,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:49:40,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:49:40,409 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:41,413 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:41,775 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:41,812 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:49:41,812 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 with policy now |
| 2025-01-17 16:49:41,815 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:49:41,816 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:49:41,816 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:49:41,817 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:49:41,976 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:49:41,977 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 |
| 2025-01-17 16:49:41,977 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:49:42,032 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/y08gbxjb-media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 |
| 2025-01-17 16:49:42,289 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/8nx8lu7v-media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 |
| 2025-01-17 16:49:42,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:42,977 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:43,410 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:43,977 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:44,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:45,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:46,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:46,978 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:47,423 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:47,505 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:48,413 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:49,415 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:50,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:50,694 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:49:50,695 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:49:50,700 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:49:50,980 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:50,980 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:49:50,980 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:49:51,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:52,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:52,725 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:52,980 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:53,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:54,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:55,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:49:55,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:49:55,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:56,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:56,982 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:49:57,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:58,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:49:58,497 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:49:59,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:00,417 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:00,983 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:01,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:02,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:03,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:04,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:04,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:05,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:06,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:07,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:08,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:09,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:09,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:10,004 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:50:10,365 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:50:10,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:50:10,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:11,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:12,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:13,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:14,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:14,466 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:15,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:16,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:17,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:18,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:19,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:20,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:20,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:21,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:22,429 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:22,991 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:23,116 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:50:23,116 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 with policy now |
| 2025-01-17 16:50:23,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:50:23,129 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:50:23,129 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:50:23,129 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:50:23,333 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/nxw300ah-media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 |
| 2025-01-17 16:50:23,422 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:23,991 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:23,992 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:50:23,992 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 |
| 2025-01-17 16:50:23,992 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:50:24,274 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/cssruwnn-media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 |
| 2025-01-17 16:50:24,434 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:24,992 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:25,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:50:25,371 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:50:25,433 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:25,451 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:26,434 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:27,437 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:28,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:28,993 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:29,461 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:30,450 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:30,994 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:31,386 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:31,456 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:32,450 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:33,459 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:33,744 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:50:33,744 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:50:33,746 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:50:33,995 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:50:33,995 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:50:34,297 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/fqs8jpnx-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:50:34,456 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:34,996 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:35,451 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:36,392 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:36,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:36,996 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:37,451 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:38,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:38,997 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:39,451 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:40,005 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:50:40,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:50:40,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:50:40,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:41,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:41,468 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:42,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:43,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:44,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:45,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:46,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:47,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:47,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:48,452 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:49,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:50,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:51,001 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:51,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:52,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:52,707 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:53,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:54,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:55,003 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:50:55,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:50:55,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:50:55,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:56,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:57,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:58,306 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:50:58,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:50:59,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:00,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:01,005 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:01,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:02,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:03,382 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:03,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:04,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:05,007 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:05,449 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:51:05,449 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:51:05,450 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 with policy now |
| 2025-01-17 16:51:05,450 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:51:05,450 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:51:05,450 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:51:05,464 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:05,667 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/9tn03h2u-media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 |
| 2025-01-17 16:51:06,007 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:51:06,007 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 |
| 2025-01-17 16:51:06,007 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:51:06,288 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/xyfths7t-media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 |
| 2025-01-17 16:51:06,459 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:07,007 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:07,470 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:08,473 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:08,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:09,486 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:10,005 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:51:10,371 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:51:10,371 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:51:10,476 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:11,009 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:11,469 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:12,467 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:13,010 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:13,479 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:14,089 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:14,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:14,797 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:51:14,798 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:51:14,798 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:51:15,010 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:51:15,011 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:51:15,481 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:16,484 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:17,011 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:17,486 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:18,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:19,491 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:19,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:20,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:21,013 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:21,517 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:22,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:23,014 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:23,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:24,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:24,754 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:25,368 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:51:25,368 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:51:25,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:26,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:27,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:28,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:29,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:30,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:30,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:31,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:32,490 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:33,490 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:34,491 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:35,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:35,491 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:36,491 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:37,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:38,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:39,020 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:39,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:40,006 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:51:40,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:51:40,369 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:51:40,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:40,500 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:41,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:42,492 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:43,022 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:43,533 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:44,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:45,023 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:45,493 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:46,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:46,495 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:47,023 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:47,495 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:48,495 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:49,024 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:49,504 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:49,947 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:51:49,947 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 with policy now |
| 2025-01-17 16:51:49,962 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:51:49,963 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:51:49,964 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:51:49,964 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:51:50,025 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:51:50,025 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 |
| 2025-01-17 16:51:50,025 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:51:50,179 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/y0g858nv-media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 |
| 2025-01-17 16:51:50,474 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/tj7hs1dl-media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 |
| 2025-01-17 16:51:50,505 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:51,025 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:51,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:51,508 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:52,025 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:52,523 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:53,503 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:54,499 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:55,026 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:55,391 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:51:55,391 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:51:55,532 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:56,512 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:56,530 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:51:57,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:58,521 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:51:59,028 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:51:59,518 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:00,524 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:01,517 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:02,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:02,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:02,857 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:52:02,858 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:52:02,858 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:52:03,029 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:52:03,030 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:52:03,279 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/fjlq00oo-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:52:03,524 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:04,548 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:05,030 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:05,521 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:06,543 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:07,525 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:07,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:08,530 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:09,032 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:09,529 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:10,006 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:52:10,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:52:10,376 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:52:10,526 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:11,032 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:11,526 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:12,526 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:13,033 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:13,386 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:13,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:14,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:15,034 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:15,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:16,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:17,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:18,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:18,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:19,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:20,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:21,037 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:21,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:22,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:23,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:24,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:24,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:25,375 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:52:25,375 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:52:25,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:26,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:27,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:28,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:29,489 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:29,528 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:30,529 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:31,529 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:32,529 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:33,529 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:34,530 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:35,406 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:35,530 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:36,530 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:37,550 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:38,460 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:52:38,461 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 with policy now |
| 2025-01-17 16:52:38,481 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:52:38,482 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:52:38,482 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:52:38,482 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:52:38,552 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:38,694 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/24avhy9l-media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 |
| 2025-01-17 16:52:39,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:39,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:52:39,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 |
| 2025-01-17 16:52:39,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:52:39,312 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/ixznevse-media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 |
| 2025-01-17 16:52:39,565 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:40,017 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:52:40,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:40,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:52:40,388 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:52:40,472 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:40,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:41,044 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:41,558 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:42,553 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:43,045 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:43,557 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:44,568 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:45,553 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:46,415 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:46,558 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:47,047 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:47,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:47,912 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:52:47,913 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:52:47,914 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:52:48,047 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:52:48,047 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:52:48,578 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:49,047 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:49,563 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:50,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:51,048 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:51,434 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:51,569 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:52,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:53,049 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:53,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:54,577 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:55,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:52:55,378 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:52:55,572 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:56,572 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:57,050 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:52:57,433 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:52:57,572 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:58,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:52:59,573 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:00,573 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:01,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:02,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:03,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:03,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:04,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:05,053 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:05,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:06,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:07,054 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:07,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:08,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:08,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:09,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:10,007 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:53:10,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:53:10,378 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:53:10,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:11,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:12,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:13,057 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:13,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:13,934 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:14,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:15,057 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:15,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:16,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:17,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:18,576 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:19,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:19,577 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:20,577 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:21,577 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:22,588 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:23,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:23,481 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:53:23,481 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_e267612638c95396263c.mp4 with policy now |
| 2025-01-17 16:53:23,488 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:53:23,489 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:53:23,489 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:53:23,489 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:53:23,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:23,701 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/w352affo-media/videos/eval/8_eval_8_e267612638c95396263c.mp4 |
| 2025-01-17 16:53:24,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:24,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:53:24,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/8_eval_8_e267612638c95396263c.mp4 |
| 2025-01-17 16:53:24,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:53:24,331 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/19jf6ls7-media/videos/eval/8_eval_8_e267612638c95396263c.mp4 |
| 2025-01-17 16:53:24,534 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:24,599 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:25,061 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:25,392 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:53:25,392 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:53:25,599 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:26,062 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:26,615 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:27,605 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:28,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:29,063 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:29,619 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:30,167 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:30,589 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:31,064 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:31,606 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:31,968 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:53:31,968 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:53:31,969 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:53:32,064 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:53:32,064 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:53:32,316 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/jf37f8xz-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:53:32,597 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:33,064 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:33,618 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:34,609 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:35,065 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:35,424 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:35,616 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:36,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:37,066 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:37,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:38,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:39,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:40,007 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:53:40,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:53:40,378 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:53:40,469 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:40,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:41,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:42,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:43,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:44,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:45,612 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:46,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:46,633 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:47,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:48,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:49,071 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:53:49,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:50,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:51,418 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:51,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:52,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:53,613 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:54,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:55,378 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:53:55,378 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:53:55,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:56,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:53:56,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:57,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:58,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:53:59,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:00,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:01,614 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:02,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:02,615 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:03,629 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:04,069 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:54:04,069 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 with policy now |
| 2025-01-17 16:54:04,076 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 |
| 2025-01-17 16:54:04,077 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:54:04,084 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:54:04,085 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:54:04,085 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:54:04,085 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:54:04,283 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/8ufhwn9p-media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 |
| 2025-01-17 16:54:04,545 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/za9mejs9-media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 |
| 2025-01-17 16:54:04,645 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:05,077 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:05,077 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:54:05,624 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:06,077 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:06,659 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:07,469 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:07,628 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:08,654 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:09,078 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:09,649 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:10,008 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:54:10,388 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:54:10,389 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:54:10,655 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:11,653 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:12,654 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:12,937 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:13,080 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:13,686 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:14,672 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:15,682 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:16,680 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:17,030 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:54:17,030 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:54:17,030 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:54:17,082 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:17,082 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:54:17,082 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:54:17,679 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:18,676 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:18,741 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:19,082 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:19,671 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:20,669 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:21,660 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:22,660 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:23,084 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:23,660 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:24,201 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:24,660 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:25,381 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:54:25,381 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:54:25,660 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:26,661 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:27,661 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:28,661 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:29,439 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:29,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:30,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:31,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:32,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:33,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:34,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:35,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:35,662 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:36,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:37,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:38,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:39,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:40,008 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:54:40,382 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:54:40,382 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:54:40,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:40,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:41,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:42,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:43,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:44,664 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:45,664 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:46,423 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:46,664 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:47,664 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:48,680 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:49,094 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:49,325 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:54:49,326 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 with policy now |
| 2025-01-17 16:54:49,353 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:54:49,353 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:54:49,353 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:54:49,353 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:54:49,547 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/cx20ngim-media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 |
| 2025-01-17 16:54:49,688 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:50,095 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:50,095 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:54:50,095 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 |
| 2025-01-17 16:54:50,095 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:54:50,366 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/nc2cb5ds-media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 |
| 2025-01-17 16:54:50,687 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:51,095 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:51,447 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:51,703 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:52,723 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:53,741 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:54,720 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:55,097 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:55,389 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:54:55,389 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:54:55,718 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:56,714 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:57,097 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:57,173 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:54:57,708 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:58,708 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:54:59,084 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:54:59,084 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:54:59,085 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:54:59,098 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:54:59,098 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:54:59,099 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:54:59,353 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/nkxm5cxn-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:54:59,722 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:00,725 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:01,099 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:01,715 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:02,419 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:02,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:03,100 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:03,737 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:04,745 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:05,771 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:06,744 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:07,443 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:07,738 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:08,744 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:09,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:10,009 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:55:10,382 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:55:10,382 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:55:10,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:11,103 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:11,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:12,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:12,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:13,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:14,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:15,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:16,728 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:17,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:18,436 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:18,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:19,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:20,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:21,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:22,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:23,436 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:23,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:24,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:25,382 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:55:25,382 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:55:25,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:26,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:27,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:28,542 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:28,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:29,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:30,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:31,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:32,730 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:33,731 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:34,436 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:34,731 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:35,731 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:36,748 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:37,113 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:37,579 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:55:37,579 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 with policy now |
| 2025-01-17 16:55:37,585 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:55:37,585 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:55:37,585 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:55:37,586 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:55:37,743 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:37,801 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/bp1jgb7k-media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 |
| 2025-01-17 16:55:38,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:38,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:55:38,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 |
| 2025-01-17 16:55:38,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:55:38,386 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/vmbhvtw0-media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 |
| 2025-01-17 16:55:38,749 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:39,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:39,643 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:39,760 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:40,009 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:55:40,114 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:40,401 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:55:40,402 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:55:40,782 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:41,766 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:42,771 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:43,116 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:43,751 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:44,746 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:45,116 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:45,451 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:45,776 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:46,767 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:47,779 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:48,786 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:49,797 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:50,810 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:51,447 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:51,799 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:52,119 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:55:52,150 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:55:52,150 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:55:52,150 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:55:52,802 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:53,119 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:53,120 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:55:53,803 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:54,804 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:55,120 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:55,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:55:55,397 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:55:55,810 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:56,830 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:56,957 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:55:57,121 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:55:57,845 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:58,832 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:55:59,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:00,830 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:01,833 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:02,328 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:02,824 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:03,124 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:03,824 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:04,824 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:05,125 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:05,824 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:06,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:07,449 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:07,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:08,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:09,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:10,010 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:56:10,391 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:56:10,391 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:56:10,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:11,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:12,469 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:12,825 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:13,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:14,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:15,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:16,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:17,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:18,450 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:18,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:19,826 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:20,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:21,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:22,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:23,450 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:23,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:24,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:25,391 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:56:25,391 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:56:25,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:26,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:27,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:28,456 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:28,827 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:29,834 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:30,879 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:30,964 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:56:30,964 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 with policy now |
| 2025-01-17 16:56:30,974 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:56:30,975 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:56:30,975 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:56:30,976 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:56:31,137 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:31,137 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:56:31,137 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 |
| 2025-01-17 16:56:31,137 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:56:31,193 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/dtoa4cr7-media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 |
| 2025-01-17 16:56:31,458 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/us3w3ip3-media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 |
| 2025-01-17 16:56:31,868 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:32,137 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:32,877 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:33,138 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:33,861 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:34,044 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:34,868 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:35,138 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:35,861 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:36,865 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:37,139 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:37,889 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:38,867 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:39,140 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:39,889 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:39,903 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:40,010 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:56:40,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:56:40,420 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:56:40,874 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:41,891 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:42,875 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:43,142 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:56:43,211 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:56:43,211 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:56:43,211 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:56:43,885 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:44,142 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:56:44,884 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:45,037 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:45,143 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:45,890 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:46,894 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:47,894 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:48,904 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:49,144 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:49,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:50,169 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:50,934 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:51,926 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:52,921 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:53,919 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:54,914 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:55,148 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:56:55,170 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:56:55,393 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:56:55,393 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:56:55,914 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:56,914 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:57,915 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:58,915 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:56:59,915 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:00,459 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:00,915 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:01,916 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:02,916 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:03,916 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:04,916 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:05,460 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:05,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:06,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:07,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:08,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:09,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:10,011 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:57:10,393 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:57:10,394 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:57:10,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:10,917 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:11,155 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:11,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:12,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:13,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:14,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:15,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:16,460 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:16,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:17,918 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:18,919 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:19,933 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:20,930 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:21,159 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:21,350 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:57:21,350 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 with policy now |
| 2025-01-17 16:57:21,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:57:21,369 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:57:21,370 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:57:21,370 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:57:21,463 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:21,584 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/j46l0rnu-media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 |
| 2025-01-17 16:57:21,939 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:22,159 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:22,160 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:57:22,160 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 |
| 2025-01-17 16:57:22,160 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:57:22,446 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/omdwopzb-media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 |
| 2025-01-17 16:57:22,970 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:23,160 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:23,995 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:24,966 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:25,403 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:57:25,404 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:57:25,960 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:26,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:26,971 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:27,161 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:27,979 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:28,960 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:29,162 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:29,985 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:30,979 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:31,560 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:32,014 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:32,985 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:33,996 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:34,989 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:35,164 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:35,165 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:57:35,275 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:57:35,275 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:57:35,275 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:57:35,407 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/irn4gesg-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:57:35,998 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:36,165 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:57:36,625 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:37,013 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:37,165 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:38,010 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:38,994 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:40,011 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:40,011 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:57:40,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:57:40,422 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:57:41,013 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:41,167 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:42,009 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:42,525 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:43,017 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:44,048 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:45,044 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:46,042 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:47,047 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:48,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:48,300 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:49,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:50,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:50,171 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:57:51,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:52,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:53,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:53,484 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:54,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:55,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:55,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:57:55,397 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:57:56,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:57,021 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:58,022 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:57:58,496 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:57:59,022 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:00,022 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:01,022 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:02,022 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:03,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:04,024 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:04,484 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:05,024 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:06,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:07,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:08,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:09,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:09,484 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:10,012 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:58:10,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:10,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:58:10,398 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:58:11,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:12,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:13,025 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:14,057 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:14,839 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:15,052 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:15,607 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:58:15,607 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 with policy now |
| 2025-01-17 16:58:15,622 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:58:15,622 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:58:15,623 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:58:15,623 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:58:15,857 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/63mgvpi5-media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 |
| 2025-01-17 16:58:16,055 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:16,181 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:16,181 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:58:16,181 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 |
| 2025-01-17 16:58:16,182 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:58:16,448 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/etdtlwhg-media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 |
| 2025-01-17 16:58:17,071 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:18,083 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:18,182 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:19,045 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:20,056 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:20,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:21,059 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:22,080 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:22,184 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:23,089 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:24,083 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:24,184 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:24,185 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:58:24,338 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:58:24,338 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:58:24,344 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:58:25,111 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:25,185 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:58:25,185 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:58:25,428 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:58:25,428 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:58:25,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:26,090 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:26,185 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:27,114 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:28,141 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:28,186 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:29,107 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:30,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:30,187 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:30,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:31,114 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:32,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:33,135 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:34,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:35,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:36,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:36,190 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:36,490 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:37,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:38,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:39,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:40,012 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:58:40,121 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:40,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:58:40,416 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:58:41,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:41,574 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:42,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:43,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:44,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:45,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:46,122 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:47,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:47,490 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:48,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:49,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:50,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:51,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:52,123 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:52,491 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:53,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:54,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:55,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:55,416 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:58:55,416 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:58:56,125 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:57,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:57,729 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:58:58,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:58:58,199 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:58:59,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:00,124 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:01,172 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:02,065 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 16:59:02,065 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 with policy now |
| 2025-01-17 16:59:02,074 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 16:59:02,074 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:59:02,075 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:59:02,075 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:59:02,142 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:02,201 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:02,201 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:59:02,201 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 |
| 2025-01-17 16:59:02,201 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 16:59:02,297 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/gkzd4227-media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 |
| 2025-01-17 16:59:02,573 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/4a0tlc3h-media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 |
| 2025-01-17 16:59:03,130 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:03,144 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:04,176 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:04,201 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:05,143 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:06,149 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:07,147 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:08,167 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:08,203 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:08,785 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:09,148 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:10,012 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:59:10,154 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:10,435 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:59:10,436 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:59:11,166 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:12,201 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:12,205 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:13,199 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:14,179 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:14,541 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:15,202 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:16,198 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:17,204 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:18,206 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:19,203 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:19,406 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 16:59:19,406 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 16:59:19,406 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 16:59:19,546 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:20,208 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 16:59:20,208 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 16:59:20,217 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:21,209 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:22,209 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:22,223 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:23,224 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:24,217 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:24,930 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:25,226 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:25,468 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:59:25,468 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:59:26,210 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:26,221 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:27,241 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:28,233 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:29,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:30,255 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:30,501 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:31,279 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:32,306 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:33,290 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:34,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:35,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:36,215 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 16:59:36,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:36,497 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:37,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:38,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:39,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:40,013 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 16:59:40,262 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:40,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:59:40,421 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:59:41,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:41,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:42,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:43,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:44,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:45,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:46,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:47,263 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:47,498 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:48,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:49,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:50,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:51,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:52,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:52,498 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:53,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:54,264 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:55,266 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:55,421 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 16:59:55,421 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 16:59:56,266 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:57,265 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:57,573 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 16:59:58,266 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 16:59:59,266 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:00,266 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:01,276 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:01,633 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:00:01,633 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 with policy now |
| 2025-01-17 17:00:01,636 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:00:01,636 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:00:01,636 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:00:01,637 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:00:01,898 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/d1jhp8kq-media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 |
| 2025-01-17 17:00:02,226 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:02,226 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:00:02,226 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 |
| 2025-01-17 17:00:02,226 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 17:00:02,287 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:02,549 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/14k53eqc-media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 |
| 2025-01-17 17:00:02,711 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:03,305 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:04,227 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:04,299 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:05,297 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:06,295 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:07,303 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:08,228 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:08,280 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:08,525 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:09,276 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:10,013 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:00:10,229 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:10,298 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:10,432 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:00:10,432 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:00:11,364 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:12,316 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:13,230 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:00:13,321 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:13,474 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:00:13,474 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:00:13,474 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:00:13,518 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/4th74f32-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:00:13,908 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:14,231 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:14,231 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:00:14,313 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:15,353 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:16,232 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:16,315 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:17,315 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:18,325 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:19,193 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:19,333 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:20,233 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:20,327 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:21,322 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:22,325 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:23,351 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:24,324 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:24,518 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:25,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:25,454 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:00:25,455 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:00:26,346 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:27,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:28,237 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:28,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:29,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:29,712 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:30,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:31,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:32,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:33,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:34,348 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:35,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:35,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:36,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:37,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:38,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:39,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:40,016 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:00:40,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:40,423 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:00:40,423 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:00:40,548 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:41,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:42,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:43,349 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:44,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:45,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:46,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:46,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:47,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:48,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:49,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:50,351 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:51,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:51,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:52,350 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:53,351 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:54,367 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:54,888 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:00:54,888 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 with policy now |
| 2025-01-17 17:00:54,900 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:00:54,900 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:00:54,900 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:00:54,900 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:00:55,106 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/avuoijqf-media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 |
| 2025-01-17 17:00:55,248 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:00:55,249 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 |
| 2025-01-17 17:00:55,249 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 17:00:55,361 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:55,439 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:00:55,440 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:00:55,540 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/u06i7q3u-media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 |
| 2025-01-17 17:00:56,249 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:56,379 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:56,550 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:00:57,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:58,250 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:00:58,369 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:00:59,422 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:00,251 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:00,386 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:01,412 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:01,995 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:02,380 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:02,532 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:01:02,532 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:01:02,532 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:01:03,252 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:01:03,252 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:01:03,398 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:04,252 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:04,428 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:05,393 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:06,253 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:06,407 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:07,015 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:07,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:08,254 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:08,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:09,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:10,014 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:01:10,255 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:10,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:10,425 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:01:10,425 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:01:11,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:12,395 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:12,559 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:13,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:14,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:15,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:16,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:17,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:18,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:18,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:19,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:20,396 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:21,398 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:22,397 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:23,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:23,527 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:24,398 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:25,398 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:25,425 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:01:25,425 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:01:26,398 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:27,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:28,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:28,569 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:29,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:30,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:31,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:32,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:33,399 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:33,938 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:34,453 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:35,420 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:35,493 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:01:35,493 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 with policy now |
| 2025-01-17 17:01:35,499 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:01:35,499 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:01:35,499 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:01:35,499 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:01:35,715 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/nio0ylyi-media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 |
| 2025-01-17 17:01:36,267 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:36,267 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:01:36,267 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 |
| 2025-01-17 17:01:36,267 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 17:01:36,433 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:36,542 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/4qxbzciu-media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 |
| 2025-01-17 17:01:37,267 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:37,438 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:38,268 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:38,439 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:39,230 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:39,455 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:40,015 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:01:40,481 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:40,482 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:01:40,483 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:01:41,475 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:42,269 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:42,469 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:43,270 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:01:43,470 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:43,582 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:01:43,582 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:01:43,582 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:01:44,241 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:44,270 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:44,271 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:01:44,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:45,455 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:46,271 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:46,478 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:47,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:48,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:49,499 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:49,649 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:50,273 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:01:50,482 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:51,494 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:52,507 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:53,514 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:54,513 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:55,439 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:01:55,439 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:01:55,509 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:01:55,561 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:56,539 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:57,589 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:58,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:01:59,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:00,278 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:00,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:00,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:01,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:02,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:03,519 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:04,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:05,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:05,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:06,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:07,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:08,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:09,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:10,015 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:02:10,436 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:02:10,436 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:02:10,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:10,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:11,520 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:12,521 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:13,521 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:14,521 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:15,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:15,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:16,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:17,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:18,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:19,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:20,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:20,554 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:21,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:22,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:23,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:24,288 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:24,522 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:25,474 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:02:25,474 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:02:25,549 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:25,599 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:26,289 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:26,293 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 |
| 2025-01-17 17:02:26,294 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval |
| 2025-01-17 17:02:26,330 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:02:26,330 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 with policy now |
| 2025-01-17 17:02:26,352 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:02:26,353 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:02:26,353 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:26,353 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:26,540 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:26,558 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/gg2zk55p-media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 |
| 2025-01-17 17:02:27,290 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:02:27,290 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 |
| 2025-01-17 17:02:27,558 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:27,577 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/atd9dbg0-media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 |
| 2025-01-17 17:02:28,290 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:28,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:29,291 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:29,579 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:30,585 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:31,584 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:31,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:32,292 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:32,587 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:33,601 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:34,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:35,601 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:36,294 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:36,597 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:36,605 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:37,294 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:02:37,589 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:37,648 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:02:37,649 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:37,650 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:38,295 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:38,295 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:02:38,584 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 17:02:38,585 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 17:02:38,625 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:38,846 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:02:39,050 INFO wandb-upload_3:2525727 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp3g4xnow0 |
| 2025-01-17 17:02:39,295 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json |
| 2025-01-17 17:02:39,295 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media |
| 2025-01-17 17:02:39,296 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table |
| 2025-01-17 17:02:39,296 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval |
| 2025-01-17 17:02:39,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:39,598 INFO SenderThread:2525727 [sender.py:send_request_log_artifact():1525] logged artifact run-ekswld3y-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTM2NDg1Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTY3ODE1OQ==', 'latestArtifact': None}} |
| 2025-01-17 17:02:39,598 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:02:39,598 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json with policy now |
| 2025-01-17 17:02:39,599 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:02:39,599 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:39,599 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:39,794 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/s3qctrm3-media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json |
| 2025-01-17 17:02:40,016 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:02:40,052 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/5lt1fzka-media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json |
| 2025-01-17 17:02:40,296 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:02:40,446 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:02:40,446 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:02:40,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:41,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:42,580 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:42,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:43,487 DEBUG SenderThread:2525727 [sender.py:send():379] send: telemetry |
| 2025-01-17 17:02:43,533 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,533 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,533 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,534 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,534 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,534 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,534 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,535 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,535 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,535 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,535 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,536 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,536 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,536 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,537 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,537 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,537 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,537 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,538 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,538 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,538 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:43,538 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:43,581 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:43,620 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 17:02:43,620 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 17:02:43,661 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 17:02:44,043 INFO wandb-upload_1:2525727 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdzc_yvl2 |
| 2025-01-17 17:02:44,298 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:02:44,298 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:02:44,298 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json |
| 2025-01-17 17:02:44,298 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval |
| 2025-01-17 17:02:44,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:44,608 INFO SenderThread:2525727 [sender.py:send_request_log_artifact():1525] logged artifact run-ekswld3y-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTM2NDkxNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTY3ODE2Nw==', 'latestArtifact': None}} |
| 2025-01-17 17:02:44,608 DEBUG SenderThread:2525727 [sender.py:send():379] send: files |
| 2025-01-17 17:02:44,608 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json with policy now |
| 2025-01-17 17:02:44,609 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:02:44,609 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:02:44,609 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:02:44,857 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/c6mxx387-events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:02:44,864 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/471qf9ez-media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json |
| 2025-01-17 17:02:45,156 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /tmp/tmp7bwae50jwandb/drdvtupu-media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json |
| 2025-01-17 17:02:45,298 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:02:45,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:46,299 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:02:46,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:47,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:48,578 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:48,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:49,301 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/config.yaml |
| 2025-01-17 17:02:49,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:50,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:51,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:52,582 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:53,583 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:54,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:54,583 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:55,446 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 17:02:55,446 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 17:02:55,583 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:56,583 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:57,583 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:58,585 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:02:59,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:02:59,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:00,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:01,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:02,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:03,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:04,575 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:03:04,586 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:05,557 DEBUG SenderThread:2525727 [sender.py:send():379] send: exit |
| 2025-01-17 17:03:05,557 INFO SenderThread:2525727 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-17 17:03:05,558 INFO SenderThread:2525727 [sender.py:send_exit():588] handling runtime: 1015 |
| 2025-01-17 17:03:05,558 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:03:05,558 INFO SenderThread:2525727 [sender.py:send_exit():594] send defer |
| 2025-01-17 17:03:05,558 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:05,558 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-17 17:03:05,558 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:05,558 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-17 17:03:05,558 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-17 17:03:05,559 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:05,559 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-17 17:03:05,559 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:05,559 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-17 17:03:05,559 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-17 17:03:05,559 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:05,559 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-17 17:03:05,559 INFO HandlerThread:2525727 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-17 17:03:05,559 DEBUG SystemMonitor:2525727 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-17 17:03:05,560 DEBUG SystemMonitor:2525727 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-17 17:03:05,559 INFO HandlerThread:2525727 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-17 17:03:05,560 INFO HandlerThread:2525727 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-17 17:03:05,614 INFO HandlerThread:2525727 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-17 17:03:05,614 INFO HandlerThread:2525727 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-17 17:03:05,614 INFO HandlerThread:2525727 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-17 17:03:05,615 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:05,615 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-17 17:03:05,615 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-17 17:03:05,615 DEBUG SenderThread:2525727 [sender.py:send():379] send: stats |
| 2025-01-17 17:03:05,615 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:05,615 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-17 17:03:05,615 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:05,616 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-17 17:03:05,616 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-17 17:03:05,616 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:05,616 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-17 17:03:06,309 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:03:08,310 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:03:11,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:11,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:03:11,663 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-17 17:03:11,664 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:11,664 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-17 17:03:11,664 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-17 17:03:11,665 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:11,665 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:11,665 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-17 17:03:11,670 DEBUG SenderThread:2525727 [sender.py:send():379] send: history |
| 2025-01-17 17:03:11,670 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 17:03:11,671 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:03:11,671 DEBUG SenderThread:2525727 [sender.py:send():379] send: summary |
| 2025-01-17 17:03:11,672 INFO SenderThread:2525727 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 17:03:11,672 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:11,672 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-17 17:03:11,672 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-17 17:03:11,672 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:11,672 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-17 17:03:11,672 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:11,672 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-17 17:03:11,673 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-17 17:03:11,673 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 17:03:11,673 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:11,673 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-17 17:03:11,673 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:11,673 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-17 17:03:12,312 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:03:12,559 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:14,365 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-17 17:03:14,365 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:14,366 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:14,366 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-17 17:03:14,366 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:14,366 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-17 17:03:14,366 INFO SenderThread:2525727 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-17 17:03:14,369 INFO SenderThread:2525727 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-17 17:03:14,372 INFO SenderThread:2525727 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-17 17:03:14,393 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-17 17:03:14,393 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:14,393 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-17 17:03:14,393 DEBUG SenderThread:2525727 [sender.py:send():379] send: artifact |
| 2025-01-17 17:03:14,559 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:15,141 INFO wandb-upload_2:2525727 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpk1ax2jia |
| 2025-01-17 17:03:15,240 INFO wandb-upload_3:2525727 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpauz4jw9z |
| 2025-01-17 17:03:15,314 INFO Thread-12 :2525727 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:03:15,351 INFO wandb-upload_0:2525727 [upload_job.py:push():88] Uploaded file /tmp/tmpaihw6o2u/wandb-job.json |
| 2025-01-17 17:03:15,915 INFO SenderThread:2525727 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ0OTM2NTUwOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ0OTIyOTkxNQ==', 'versionIndex': 443}}} |
| 2025-01-17 17:03:15,916 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:15,916 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-17 17:03:15,916 INFO SenderThread:2525727 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/config.yaml config.yaml |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json wandb-summary.json |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/requirements.txt requirements.txt |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-17 17:03:16,315 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 events.out.tfevents.1737150370.leonmkim-ROG-Strix-G15CS-G15CS.2525562.0 |
| 2025-01-17 17:03:16,319 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 17:03:16,319 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log output.log |
| 2025-01-17 17:03:16,320 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/diff.patch diff.patch |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 media/videos/eval/13_eval_13_c6d655bdee80c0076f89.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 media/videos/eval/6_eval_6_8987b2b58d34cfe8cb42.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 media/videos/eval/18_eval_18_a64238a986bd4b5570bb.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 media/videos/eval/2_eval_2_fe77002b2cb4622b4b22.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 media/videos/eval/11_eval_11_9995a807c7d1da4dc372.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 media/videos/eval/10_eval_10_d62c4c83fb8fd6ab16b2.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 media/videos/eval/15_eval_15_bf32b829657f67b2af7f.mp4 |
| 2025-01-17 17:03:16,322 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 media/videos/eval/12_eval_12_fa0d41412850078538dc.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 media/videos/eval/5_eval_5_7804d187012a10ce94dd.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 media/videos/eval/0_eval_0_dc7e93c4374b379f9dcb.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 media/videos/eval/4_eval_4_dc6eafd2d4d9085da58f.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 media/videos/eval/14_eval_14_7b103374eb4cd2abc762.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 media/videos/eval/3_eval_3_27cc3d32f9646ff9f47c.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 media/videos/eval/1_eval_1_c6ec0e759aa9c9feadea.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 media/videos/eval/19_eval_19_c86b20bc67f17498d176.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 media/videos/eval/16_eval_16_8141160245b4d1bfae0d.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 media/videos/eval/17_eval_17_485ca0c6da25b819c2e9.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/8_eval_8_e267612638c95396263c.mp4 media/videos/eval/8_eval_8_e267612638c95396263c.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 media/videos/eval/7_eval_7_b37c7107611e8b218c9d.mp4 |
| 2025-01-17 17:03:16,323 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 media/videos/eval/9_eval_9_6bc03923ced705dbe34b.mp4 |
| 2025-01-17 17:03:16,324 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json media/table/eval/success_rate_ci_20_626687eda62aee1b0486.table.json |
| 2025-01-17 17:03:16,324 INFO SenderThread:2525727 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json media/table/eval/total_success_rate_ci_21_8b681d6b013e3fd9f64a.table.json |
| 2025-01-17 17:03:16,324 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-17 17:03:16,324 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:16,324 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:16,324 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-17 17:03:16,324 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:16,324 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-17 17:03:16,324 INFO SenderThread:2525727 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 17:03:16,505 INFO wandb-upload_1:2525727 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/config.yaml |
| 2025-01-17 17:03:16,508 INFO wandb-upload_2:2525727 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/wandb-summary.json |
| 2025-01-17 17:03:16,536 INFO wandb-upload_3:2525727 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/requirements.txt |
| 2025-01-17 17:03:16,560 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:16,561 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:16,567 INFO wandb-upload_0:2525727 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/files/output.log |
| 2025-01-17 17:03:16,767 INFO Thread-11 (_thread_body):2525727 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-17 17:03:16,767 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:16,767 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-17 17:03:16,768 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:16,768 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-17 17:03:16,768 INFO SenderThread:2525727 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-17 17:03:16,771 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-17 17:03:16,771 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:16,771 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-17 17:03:16,772 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:16,772 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-17 17:03:16,772 INFO SenderThread:2525727 [file_stream.py:finish():601] file stream finish called |
| 2025-01-17 17:03:17,132 INFO SenderThread:2525727 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-17 17:03:17,132 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-17 17:03:17,133 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:17,133 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-17 17:03:17,133 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:17,133 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-17 17:03:17,133 INFO SenderThread:2525727 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-17 17:03:17,133 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 17:03:17,133 INFO HandlerThread:2525727 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-17 17:03:17,133 DEBUG SenderThread:2525727 [sender.py:send():379] send: final |
| 2025-01-17 17:03:17,133 DEBUG SenderThread:2525727 [sender.py:send():379] send: footer |
| 2025-01-17 17:03:17,133 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 17:03:17,133 INFO SenderThread:2525727 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-17 17:03:17,134 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:17,134 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-17 17:03:17,134 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:17,135 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: server_info |
| 2025-01-17 17:03:17,136 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 17:03:17,136 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-17 17:03:17,137 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-17 17:03:17,138 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 17:03:17,198 DEBUG SenderThread:2525727 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 17:03:17,199 INFO MainThread:2525727 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-17 17:03:17,199 INFO MainThread:2525727 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-17 17:03:17,199 INFO MainThread:2525727 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-17 17:03:17,200 DEBUG HandlerThread:2525727 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-17 17:03:17,200 INFO HandlerThread:2525727 [handler.py:finish():882] shutting down handler |
| 2025-01-17 17:03:18,137 INFO WriterThread:2525727 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_1/ekswld3y/wandb/run-20250117_164609-ekswld3y/run-ekswld3y.wandb |
| 2025-01-17 17:03:18,198 INFO SenderThread:2525727 [sender.py:finish():1615] shutting down sender |
| 2025-01-17 17:03:18,199 INFO SenderThread:2525727 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 17:03:18,199 INFO SenderThread:2525727 [file_pusher.py:join():175] waiting for file pusher |
|
|
