| 2025-01-15 21:45:12,624 INFO StreamThr :2470873 [internal.py:wandb_internal():85] W&B internal server running at pid: 2470873, started at: 2025-01-15 21:45:12.624488 |
| 2025-01-15 21:45:12,625 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status |
| 2025-01-15 21:45:12,637 INFO WriterThread:2470873 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/run-roatk2bk.wandb |
| 2025-01-15 21:45:12,638 DEBUG SenderThread:2470873 [sender.py:send():379] send: header |
| 2025-01-15 21:45:12,645 DEBUG SenderThread:2470873 [sender.py:send():379] send: run |
| 2025-01-15 21:45:12,893 INFO SenderThread:2470873 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files |
| 2025-01-15 21:45:12,893 INFO SenderThread:2470873 [sender.py:_start_run_threads():1188] run started: roatk2bk with start time 1736995512.624682 |
| 2025-01-15 21:45:12,912 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-15 21:45:12,912 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: check_version |
| 2025-01-15 21:45:12,986 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-15 21:45:12,990 DEBUG HandlerThread:2470873 [system_info.py:__init__():26] System info init |
| 2025-01-15 21:45:12,991 DEBUG HandlerThread:2470873 [system_info.py:__init__():41] System info init done |
| 2025-01-15 21:45:12,991 INFO HandlerThread:2470873 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-15 21:45:12,991 INFO SystemMonitor:2470873 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-15 21:45:12,991 INFO HandlerThread:2470873 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-15 21:45:12,991 INFO SystemMonitor:2470873 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-15 21:45:12,992 INFO SystemMonitor:2470873 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-15 21:45:12,992 INFO SystemMonitor:2470873 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-15 21:45:12,993 INFO SystemMonitor:2470873 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-15 21:45:12,993 INFO SystemMonitor:2470873 [interfaces.py:start():188] Started network monitoring |
| 2025-01-15 21:45:13,004 DEBUG HandlerThread:2470873 [system_info.py:probe():152] Probing system |
| 2025-01-15 21:45:13,005 DEBUG HandlerThread:2470873 [system_info.py:_probe_git():137] Probing git |
| 2025-01-15 21:45:13,010 DEBUG HandlerThread:2470873 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-15 21:45:13,010 DEBUG HandlerThread:2470873 [system_info.py:probe():200] Probing system done |
| 2025-01-15 21:45:13,010 DEBUG HandlerThread:2470873 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-16T02:45:13.004292', 'startedAt': '2025-01-16T02:45:12.616933', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3748.5323750000002, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.919, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 326.04469299316406}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-15 21:45:13,010 INFO HandlerThread:2470873 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-15 21:45:13,010 INFO HandlerThread:2470873 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-15 21:45:13,010 DEBUG HandlerThread:2470873 [system_info.py:_save_code():44] Saving code |
| 2025-01-15 21:45:13,011 DEBUG HandlerThread:2470873 [system_info.py:_save_code():67] Saving code done |
| 2025-01-15 21:45:13,011 DEBUG HandlerThread:2470873 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-15 21:45:13,109 DEBUG HandlerThread:2470873 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-15 21:45:13,109 INFO HandlerThread:2470873 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-15 21:45:13,113 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:45:13,113 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-15 21:45:13,113 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-15 21:45:13,113 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-15 21:45:13,113 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-15 21:45:13,359 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-15 21:45:13,359 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-15 21:45:13,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:45:13,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:45:13,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:13,363 INFO HandlerThread:2470873 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/tb" |
| save: true |
| } |
| control { |
| relay_id: "7e40abce871040128ee2222932bf4fd5" |
| } |
| _info { |
| stream_id: "roatk2bk" |
| } |
|
|
| 2025-01-15 21:45:13,364 DEBUG HandlerThread:2470873 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-15 21:45:13,367 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/2krs8t93-wandb-metadata.json |
| 2025-01-15 21:45:13,395 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/i5dtzjt2-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 21:45:13,398 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/5i0rcxno-diff.patch |
| 2025-01-15 21:45:13,404 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/zzqthulp-code/FISH/eval_robot.py |
| 2025-01-15 21:45:13,535 DEBUG SenderThread:2470873 [sender.py:send():379] send: telemetry |
| 2025-01-15 21:45:13,536 DEBUG SenderThread:2470873 [sender.py:send():379] send: tbrecord |
| 2025-01-15 21:45:13,536 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:45:13,536 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 with policy live |
| 2025-01-15 21:45:13,712 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/2kz91b1o-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/requirements.txt |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/diff.patch |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-metadata.json |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/code/FISH/eval_robot.py |
| 2025-01-15 21:45:13,895 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/code/FISH |
| 2025-01-15 21:45:13,896 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/code |
| 2025-01-15 21:45:14,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:15,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:15,894 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:45:16,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:17,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:18,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:18,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:19,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:20,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:21,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:22,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:23,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:23,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:24,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:25,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:26,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:27,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:28,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:45:28,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:45:28,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:28,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:29,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:30,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:31,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:32,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:33,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:34,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:34,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:35,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:36,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:37,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:38,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:39,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:39,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:40,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:41,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:41,678 DEBUG SenderThread:2470873 [sender.py:send():379] send: config |
| 2025-01-15 21:45:42,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:43,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:45:43,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:45:43,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:43,901 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:45:44,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:44,435 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:44,902 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/config.yaml |
| 2025-01-15 21:45:45,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:46,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:47,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:47,903 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:45:48,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:49,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:50,210 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:50,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:51,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:51,904 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:45:52,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:53,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:54,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:55,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:45:55,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:56,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:57,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:58,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:45:58,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:45:58,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:45:59,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:00,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:00,524 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:01,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:01,907 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:02,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:03,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:04,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:05,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:05,527 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:05,908 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:06,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:07,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:08,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:09,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:10,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:11,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:11,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:12,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:12,993 DEBUG SystemMonitor:2470873 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-15 21:46:12,994 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:46:13,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:46:13,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:46:13,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:14,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:15,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:16,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:16,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:17,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:18,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:18,469 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:46:18,470 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:46:18,470 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 with policy now |
| 2025-01-15 21:46:18,470 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:46:18,470 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:46:18,471 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:46:18,768 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/z7jz6cz9-media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 |
| 2025-01-15 21:46:18,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 |
| 2025-01-15 21:46:18,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:46:18,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media |
| 2025-01-15 21:46:18,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos |
| 2025-01-15 21:46:18,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:46:19,189 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/o7263fbs-media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 |
| 2025-01-15 21:46:19,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:19,912 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:20,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:21,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:21,954 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:22,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:23,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:23,914 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:24,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:25,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:25,914 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:26,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:27,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:27,706 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:28,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:46:28,360 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:46:28,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:29,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:30,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:31,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:32,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:33,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:33,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:34,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:35,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:36,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:37,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:38,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:38,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:38,920 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:46:39,179 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/6brbwr5a-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:46:39,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:39,920 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:40,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:41,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:41,921 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:42,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:43,003 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:46:43,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:46:43,361 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:46:43,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:43,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:43,921 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:46:44,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:45,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:46,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:47,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:48,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:49,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:49,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:50,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:51,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:52,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:53,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:54,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:46:54,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:55,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:56,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:57,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:58,361 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:46:58,361 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:46:58,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:59,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:46:59,420 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:00,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:01,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:02,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:03,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:04,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:05,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:05,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:06,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:07,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:08,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:09,376 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:10,376 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:10,404 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:10,445 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:47:10,446 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:47:10,446 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 with policy now |
| 2025-01-15 21:47:10,446 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:47:10,446 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:47:10,447 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:47:10,708 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/bgeeyww1-media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 |
| 2025-01-15 21:47:10,930 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:47:10,930 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 |
| 2025-01-15 21:47:10,930 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:47:11,202 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/kgz7uqes-media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 |
| 2025-01-15 21:47:11,377 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:11,930 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:12,377 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:12,995 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:47:13,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:47:13,368 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:47:13,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:14,380 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:15,385 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:15,932 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:16,093 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:16,385 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:17,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:17,932 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:18,380 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:19,380 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:20,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:21,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:21,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:22,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:23,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:24,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:25,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:26,381 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:26,588 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:26,936 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:47:27,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:27,810 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:47:27,811 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:47:27,811 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:47:27,936 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:27,936 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:47:28,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:47:28,363 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:47:28,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:29,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:29,937 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:30,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:31,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:31,723 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:31,937 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:32,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:33,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:34,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:35,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:36,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:37,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:37,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:38,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:39,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:40,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:41,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:42,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:42,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:42,995 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:47:43,362 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:47:43,362 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:47:43,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:44,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:45,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:46,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:47,386 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:47,478 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:47,943 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:48,113 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:47:48,113 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 with policy now |
| 2025-01-15 21:47:48,113 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:47:48,114 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:47:48,114 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:47:48,115 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:47:48,354 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/22gfzdwj-media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 |
| 2025-01-15 21:47:48,389 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:48,944 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:47:48,944 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 |
| 2025-01-15 21:47:48,944 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:47:49,217 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/yjg1aet9-media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 |
| 2025-01-15 21:47:49,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:49,944 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:50,389 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:50,945 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:51,390 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:52,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:52,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:53,397 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:53,946 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:54,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:55,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:55,947 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:47:55,947 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:56,200 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/mlrje92h-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:47:56,388 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:56,617 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:47:56,617 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:47:56,617 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:47:56,947 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:47:57,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:57,905 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:47:57,947 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:47:58,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:47:58,367 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:47:58,413 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:47:59,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:00,390 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:01,390 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:01,949 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:02,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:03,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:03,390 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:04,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:05,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:06,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:07,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:08,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:08,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:09,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:10,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:11,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:12,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:12,996 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:48:13,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:48:13,363 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:48:13,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:13,485 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:14,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:15,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:16,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:17,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:18,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:19,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:19,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:20,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:21,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:22,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:23,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:24,371 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:24,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:25,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:26,393 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:27,395 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:27,958 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:27,996 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:48:27,996 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 with policy now |
| 2025-01-15 21:48:28,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:48:28,006 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:48:28,006 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:48:28,006 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:48:28,254 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/b0rvfr7r-media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 |
| 2025-01-15 21:48:28,363 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:48:28,363 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:48:28,394 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:28,958 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:48:28,959 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 |
| 2025-01-15 21:48:28,959 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:48:29,255 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/rjnbpuoi-media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 |
| 2025-01-15 21:48:29,398 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:29,419 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:29,959 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:30,403 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:30,959 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:31,403 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:32,403 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:33,427 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:33,960 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:34,403 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:34,457 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:35,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:35,961 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:36,402 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:37,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:38,411 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:39,404 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:40,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:40,420 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:41,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:41,672 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:48:41,672 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:48:41,673 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:48:41,963 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:48:41,963 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:48:42,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:42,996 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:48:43,366 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:48:43,367 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:48:43,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:43,964 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:44,410 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:45,411 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:45,964 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:46,313 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:46,416 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:47,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:47,965 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:48:48,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:49,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:50,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:51,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:51,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:52,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:53,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:54,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:55,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:56,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:48:56,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:57,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:58,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:48:58,364 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:48:58,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:48:59,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:00,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:01,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:01,442 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:02,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:03,416 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:03,971 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:04,245 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:49:04,245 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 with policy now |
| 2025-01-15 21:49:04,254 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:49:04,255 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:49:04,255 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:49:04,255 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:49:04,411 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:04,483 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/6v0b62sc-media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 |
| 2025-01-15 21:49:04,971 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:49:04,971 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 |
| 2025-01-15 21:49:04,971 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:49:05,243 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/9pexg8i7-media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 |
| 2025-01-15 21:49:05,416 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:05,971 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:06,411 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:07,286 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:07,419 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:08,422 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:09,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:09,973 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:10,422 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:11,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:11,974 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:12,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:12,737 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:12,997 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:49:13,369 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:49:13,369 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:49:13,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:13,713 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:49:13,713 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:49:13,713 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:49:13,974 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:49:13,975 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:13,975 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:49:14,254 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/3wz60emi-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:49:14,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:15,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:15,975 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:16,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:17,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:17,976 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:18,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:18,435 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:19,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:19,977 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:20,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:21,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:22,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:23,375 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:23,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:24,421 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:25,422 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:26,422 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:27,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:28,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:49:28,365 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:49:28,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:28,428 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:29,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:30,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:31,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:32,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:33,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:34,376 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:34,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:35,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:36,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:37,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:38,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:39,376 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:39,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:40,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:41,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:42,425 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:42,997 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:49:43,364 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:49:43,365 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:49:43,425 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:44,425 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:44,510 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:45,425 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:46,435 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:47,038 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:49:47,038 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 with policy now |
| 2025-01-15 21:49:47,042 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:49:47,042 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:49:47,042 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:49:47,042 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:49:47,335 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/mhfqktbw-media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 |
| 2025-01-15 21:49:47,429 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:47,987 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:47,987 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:49:47,987 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 |
| 2025-01-15 21:49:47,987 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:49:48,334 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/xj0rb057-media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 |
| 2025-01-15 21:49:48,441 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:48,987 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:49,440 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:50,405 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:50,439 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:51,439 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:51,988 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:52,438 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:53,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:54,440 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:55,445 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:55,617 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:49:55,990 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:49:56,442 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:57,445 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:58,378 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:49:58,379 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:49:58,449 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:49:59,444 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:00,453 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:00,771 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:50:00,771 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:50:00,772 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:00,772 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:50:00,991 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:50:00,992 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:50:01,444 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:01,992 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:02,445 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:03,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:03,992 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:04,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:05,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:05,993 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:06,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:06,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:07,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:07,994 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:08,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:09,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:10,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:11,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:12,379 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:12,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:12,998 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:50:13,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:50:13,365 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:50:13,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:14,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:15,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:16,447 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:17,444 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:17,448 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:18,449 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:19,449 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:20,449 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:21,449 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:22,450 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:22,987 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:23,450 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:24,000 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:24,450 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:25,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:26,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:27,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:28,365 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:50:28,365 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:50:28,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:28,504 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:29,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:30,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:31,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:32,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:33,453 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:34,004 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:34,149 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:34,435 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:50:34,435 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 with policy now |
| 2025-01-15 21:50:34,439 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:50:34,439 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:50:34,439 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:50:34,440 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:50:34,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:34,690 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/56yq0pm1-media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 |
| 2025-01-15 21:50:35,004 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:50:35,004 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 |
| 2025-01-15 21:50:35,004 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:50:35,285 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/i8416a62-media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 |
| 2025-01-15 21:50:35,462 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:36,004 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:36,463 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:37,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:38,468 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:39,398 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:39,481 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:40,006 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:40,483 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:41,466 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:42,007 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:42,469 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:42,998 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:50:43,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:50:43,370 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:50:43,485 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:44,445 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:44,472 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:45,493 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:46,485 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:47,489 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:48,488 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:49,491 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:50,398 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:50,489 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:51,494 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:52,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:53,490 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:54,495 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:55,399 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:50:55,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:56,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:57,491 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:57,842 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:50:57,842 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:50:57,842 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:50:58,012 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:50:58,012 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:50:58,013 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:50:58,263 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/b1z9epwb-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:50:58,385 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:50:58,385 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:50:58,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:50:59,494 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:00,013 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:00,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:00,536 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:01,495 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:02,014 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:02,502 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:03,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:04,014 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:04,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:05,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:05,541 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:06,015 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:06,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:07,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:08,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:09,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:10,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:11,396 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:11,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:12,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:12,998 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:51:13,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:51:13,374 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:51:13,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:14,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:15,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:16,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:16,517 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:17,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:18,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:19,498 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:20,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:21,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:22,397 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:22,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:23,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:24,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:25,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:26,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:27,397 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:27,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:28,374 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:51:28,374 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:51:28,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:29,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:30,024 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:30,514 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:31,430 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:51:31,430 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 with policy now |
| 2025-01-15 21:51:31,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:51:31,446 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:51:31,446 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:51:31,447 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:51:31,504 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:31,676 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/hgjwvamf-media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 |
| 2025-01-15 21:51:32,025 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:32,025 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:51:32,025 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 |
| 2025-01-15 21:51:32,025 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:51:32,315 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/dxfpwmx0-media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 |
| 2025-01-15 21:51:32,492 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:32,520 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:33,520 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:34,026 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:34,525 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:35,522 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:36,027 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:36,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:37,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:37,982 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:38,541 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:39,529 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:40,029 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:40,541 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:41,543 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:41,897 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:51:41,897 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:51:41,897 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:51:42,030 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:51:42,030 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:51:42,551 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:42,999 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:51:42,999 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:43,382 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:51:43,383 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:51:43,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:44,030 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:44,543 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:45,543 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:46,549 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:47,550 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:48,032 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:48,387 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:48,548 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:49,544 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:50,033 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:51:50,544 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:51,544 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:52,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:53,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:53,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:54,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:55,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:56,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:57,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:58,378 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:51:58,378 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:51:58,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:51:58,549 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:51:59,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:00,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:01,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:02,037 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:02,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:03,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:04,410 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:04,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:05,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:06,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:07,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:08,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:09,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:10,041 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:10,410 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:10,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:11,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:12,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:12,999 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:52:13,378 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:52:13,378 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:52:13,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:14,042 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:14,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:15,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:15,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:16,043 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:16,555 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:16,815 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:52:16,815 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 with policy now |
| 2025-01-15 21:52:16,829 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:52:16,829 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:52:16,829 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:52:16,830 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:52:17,044 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:52:17,044 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 |
| 2025-01-15 21:52:17,044 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:52:17,075 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/xsm8xo6m-media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 |
| 2025-01-15 21:52:17,344 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/ctcy9ilg-media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 |
| 2025-01-15 21:52:17,561 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:18,044 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:18,575 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:19,044 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:19,561 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:20,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:20,555 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:21,554 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:22,045 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:22,569 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:23,556 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:24,046 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:24,570 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:25,477 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:25,564 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:26,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:27,565 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:28,399 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:52:28,399 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:52:28,587 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:29,572 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:30,483 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:30,584 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:31,576 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:31,960 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:52:31,960 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:52:31,961 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:52:32,049 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:52:32,050 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:32,050 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:52:32,302 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/hj7yr4f7-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:52:32,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:33,581 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:34,050 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:34,586 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:35,561 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:35,605 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:36,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:37,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:38,052 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:38,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:39,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:40,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:41,416 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:41,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:42,577 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:43,000 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:52:43,379 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:52:43,379 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:52:43,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:44,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:45,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:46,468 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:46,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:47,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:48,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:49,578 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:50,579 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:51,579 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:51,928 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:52,590 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:53,349 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:52:53,349 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 with policy now |
| 2025-01-15 21:52:53,355 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:52:53,356 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:52:53,356 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:52:53,356 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:52:53,583 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/e9k3rryo-media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 |
| 2025-01-15 21:52:53,585 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:54,058 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:54,058 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:52:54,058 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 |
| 2025-01-15 21:52:54,059 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:52:54,340 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/61hoxzze-media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 |
| 2025-01-15 21:52:54,592 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:55,058 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:55,595 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:56,597 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:57,047 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:52:57,625 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:58,060 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:52:58,399 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:52:58,400 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:52:58,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:52:59,600 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:00,617 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:01,618 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:02,058 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:02,061 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:53:02,061 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:02,612 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:03,000 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:53:03,000 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:53:03,001 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:53:03,061 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:53:03,062 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:53:03,629 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:04,062 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:04,634 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:05,619 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:06,611 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:07,141 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:07,651 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:08,063 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:08,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:09,630 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:10,643 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:11,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:12,065 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:12,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:13,000 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:53:13,002 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:13,379 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:53:13,380 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:53:13,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:14,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:15,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:16,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:17,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:18,017 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:18,067 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:18,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:19,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:20,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:21,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:22,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:23,422 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:23,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:24,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:25,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:26,624 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:27,635 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:28,071 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:28,381 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:53:28,381 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 with policy now |
| 2025-01-15 21:53:28,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:53:28,391 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:53:28,391 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:53:28,392 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:53:28,397 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:53:28,397 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:53:28,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:28,629 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/jp7098hy-media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 |
| 2025-01-15 21:53:28,630 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:29,072 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:53:29,072 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 |
| 2025-01-15 21:53:29,072 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:53:29,342 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/utk77r6u-media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 |
| 2025-01-15 21:53:29,646 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:30,072 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:30,634 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:31,073 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:31,666 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:32,642 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:33,647 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:34,074 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:34,432 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:34,649 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:35,656 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:36,075 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:36,657 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:37,044 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:53:37,044 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:53:37,045 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:53:37,075 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:53:37,075 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:53:37,325 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/9thvpq01-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:53:37,668 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:38,075 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:38,659 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:39,458 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:39,662 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:40,660 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:41,672 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:42,077 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:42,672 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:43,001 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:53:43,391 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:53:43,391 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:53:43,667 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:44,515 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:44,683 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:45,681 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:46,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:47,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:48,080 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:53:48,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:49,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:50,455 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:50,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:51,664 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:52,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:53,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:54,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:55,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:56,455 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:53:56,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:57,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:58,383 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:53:58,383 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:53:58,665 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:53:59,666 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:00,666 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:01,500 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:01,666 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:02,709 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:03,692 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:04,043 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:54:04,043 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 with policy now |
| 2025-01-15 21:54:04,061 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:54:04,062 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:54:04,062 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:54:04,062 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:54:04,086 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:04,086 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:54:04,086 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 |
| 2025-01-15 21:54:04,086 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:54:04,323 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/0alu68rb-media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 |
| 2025-01-15 21:54:04,586 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/wr03doaa-media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 |
| 2025-01-15 21:54:04,683 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:05,678 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:06,087 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:06,696 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:07,087 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:07,103 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:07,697 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:08,680 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:09,697 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:10,088 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:10,701 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:11,693 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:12,089 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:12,697 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:12,853 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:13,001 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:54:13,399 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:54:13,399 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:54:13,707 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:14,697 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:15,694 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:16,693 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:17,727 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:18,480 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:18,709 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:19,091 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:54:19,096 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:54:19,096 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:54:19,100 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:54:19,730 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:20,092 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:54:20,092 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:20,092 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:54:20,727 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:21,718 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:22,093 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:22,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:23,513 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:23,710 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:24,094 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:24,710 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:25,710 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:26,711 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:27,711 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:28,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:54:28,384 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:54:28,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:29,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:29,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:30,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:31,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:32,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:33,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:34,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:35,468 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:35,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:36,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:37,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:38,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:39,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:40,468 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:40,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:41,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:42,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:43,002 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:54:43,384 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:54:43,385 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:54:43,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:44,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:45,489 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:45,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:46,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:47,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:48,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:49,714 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:50,729 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:51,053 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:51,725 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:51,773 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:54:51,773 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:54:51,773 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 with policy now |
| 2025-01-15 21:54:51,774 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:54:51,774 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:54:51,774 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:54:52,030 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/lmglli64-media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 |
| 2025-01-15 21:54:52,105 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:54:52,105 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 |
| 2025-01-15 21:54:52,105 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:54:52,401 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/r6bjjpye-media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 |
| 2025-01-15 21:54:52,735 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:53,105 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:53,734 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:54,750 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:55,106 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:55,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:56,505 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:54:56,757 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:57,107 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:54:57,755 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:58,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:54:58,401 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:54:58,751 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:54:59,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:00,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:01,108 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:01,522 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:01,776 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:02,109 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:55:02,149 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:55:02,150 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:55:02,150 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:55:02,787 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:03,110 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:03,110 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:55:03,788 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:04,766 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:05,111 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:05,789 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:06,765 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:06,804 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:07,803 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:08,791 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:09,112 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:09,800 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:10,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:11,807 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:12,179 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:12,791 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:13,002 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:55:13,388 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:55:13,388 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:55:13,791 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:14,791 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:15,115 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:15,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:16,791 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:17,532 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:17,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:18,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:19,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:20,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:21,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:22,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:23,029 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:23,792 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:24,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:25,119 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:25,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:26,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:27,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:28,037 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:28,387 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:55:28,387 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:55:28,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:29,121 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:29,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:30,793 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:31,122 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:31,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:32,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:33,479 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:33,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:34,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:35,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:36,795 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:37,795 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:38,479 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:38,795 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:39,799 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:40,823 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:40,926 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:55:40,926 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 with policy now |
| 2025-01-15 21:55:40,935 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:55:40,935 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:55:40,936 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:55:40,936 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:55:41,126 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:41,126 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:55:41,126 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 |
| 2025-01-15 21:55:41,126 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:55:41,174 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/lgjmpjp7-media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 |
| 2025-01-15 21:55:41,428 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/tx0smn3m-media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 |
| 2025-01-15 21:55:41,827 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:42,807 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:43,003 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:55:43,126 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:43,413 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:55:43,413 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:55:43,512 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:43,829 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:44,831 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:45,820 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:46,822 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:47,128 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:47,824 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:48,522 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:48,846 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:49,886 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:50,853 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:51,130 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:55:51,882 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:52,850 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:53,885 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:54,507 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:55:54,860 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:55,889 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:56,868 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:57,877 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:58,435 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:55:58,435 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:55:58,881 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:55:59,133 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:55:59,219 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:55:59,219 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:55:59,219 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:55:59,375 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/t0pun5g6-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:55:59,880 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:00,133 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 21:56:00,133 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:56:00,274 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:00,888 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:01,134 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:01,871 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:02,906 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:03,134 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:03,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:04,873 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:05,136 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:05,753 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:05,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:06,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:07,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:08,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:09,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:10,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:11,488 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:11,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:12,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:13,005 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:56:13,139 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:13,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:56:13,401 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:56:13,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:14,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:15,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:16,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:16,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:17,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:18,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:19,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:20,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:21,142 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:21,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:21,933 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:22,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:23,143 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:23,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:24,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:25,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:26,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:27,145 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:27,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:27,926 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:28,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:56:28,401 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:56:28,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:29,876 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:30,898 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:31,888 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:31,978 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 21:56:31,978 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 with policy now |
| 2025-01-15 21:56:31,991 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:56:31,991 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 21:56:31,991 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:56:31,992 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:56:32,147 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 21:56:32,148 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 |
| 2025-01-15 21:56:32,148 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 21:56:32,225 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/tyvt3rah-media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 |
| 2025-01-15 21:56:32,492 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/hvflza3a-media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 |
| 2025-01-15 21:56:32,891 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:33,088 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:33,148 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:33,890 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:34,912 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:35,149 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:35,908 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:36,914 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:37,939 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:38,515 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:38,925 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:39,151 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:39,936 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:40,917 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:41,152 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 21:56:41,940 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:42,966 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:43,003 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:56:43,404 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:56:43,404 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:56:43,960 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:44,508 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:44,954 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:45,942 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:46,938 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:47,950 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:48,943 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:49,517 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:49,935 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:50,940 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:51,978 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:52,951 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:53,960 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:54,537 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:56:54,962 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:55,969 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:56,988 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:57,973 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:56:58,427 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:56:58,428 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:56:58,979 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:00,018 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:00,528 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:01,023 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:02,025 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:02,992 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:03,987 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:04,997 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:05,537 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:05,988 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:07,004 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:08,017 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:09,027 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:10,024 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:11,027 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:11,524 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:12,040 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:13,004 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:57:13,026 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:13,419 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:57:13,420 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:57:14,035 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:15,023 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:16,020 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:17,022 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:17,523 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:18,030 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:19,024 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:20,074 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:21,067 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:22,043 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:22,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:23,053 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:24,055 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:25,059 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:26,064 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:27,050 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:27,554 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:28,069 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:28,432 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:57:28,432 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:57:29,077 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:30,066 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:31,064 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:32,083 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:32,556 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:33,096 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:34,070 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:35,094 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:36,072 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:37,109 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:38,089 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:38,520 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:39,105 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:40,102 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:41,095 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:42,099 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:43,004 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:57:43,126 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:43,426 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:57:43,426 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:57:43,539 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:44,111 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:45,127 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:46,118 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:47,118 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:48,122 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:49,113 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:49,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:50,125 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:51,123 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:52,120 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:53,150 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:54,153 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:54,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:57:55,156 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:56,145 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:57,134 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:58,160 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:58,425 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:57:58,425 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:57:59,155 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:57:59,547 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:00,146 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:01,182 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:02,151 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:03,167 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:04,180 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:04,555 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:05,170 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:06,172 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:07,179 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:08,174 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:09,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:10,232 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:10,546 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:11,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:12,205 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:13,005 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:58:13,210 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:13,453 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:58:13,453 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:58:14,246 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:15,245 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:15,559 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:16,265 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:17,226 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:18,234 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:19,234 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:20,234 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:20,601 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:21,239 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:22,237 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:23,242 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:24,244 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:25,273 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:26,262 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:26,571 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:27,270 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:28,251 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:28,431 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:58:28,431 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:58:29,264 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:30,285 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:31,278 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:31,580 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:32,280 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:33,283 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:34,308 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:35,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:36,352 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:37,309 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:37,579 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:38,309 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:39,343 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:40,313 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:41,311 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:42,321 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:42,640 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:43,005 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:58:43,329 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:43,464 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:58:43,464 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:58:44,347 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:45,339 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:46,342 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:47,342 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:48,339 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:48,596 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:49,378 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:50,358 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:51,359 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:52,345 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:53,358 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:53,604 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:54,354 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:55,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:56,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:57,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:58,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:58:58,448 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:58:58,448 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:58:58,645 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:58:59,373 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:00,378 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:01,372 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:02,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:03,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:04,402 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:04,608 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:05,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:06,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:07,426 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:08,431 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:09,417 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:10,429 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:10,608 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:11,424 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:12,431 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:13,006 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:59:13,451 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:13,463 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:59:13,463 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:59:14,465 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:15,454 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:15,619 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:16,428 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:17,456 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:18,457 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:19,455 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:20,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:20,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:21,448 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:22,487 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:23,495 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:24,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:25,452 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:26,458 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:26,614 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:27,481 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:28,453 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:59:28,453 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:59:28,497 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:29,475 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:30,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:31,478 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:31,645 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:32,474 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:33,469 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:34,490 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:35,530 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:36,512 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:37,505 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:37,623 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:38,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:39,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:40,531 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:41,545 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:42,563 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:42,642 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:43,006 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 21:59:43,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:59:43,461 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:59:43,566 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:44,557 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:45,558 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:46,561 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:47,565 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:48,595 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:48,626 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:49,569 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:50,575 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:51,589 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:52,588 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:53,583 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:54,590 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:54,626 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:59:55,586 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:56,602 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:57,591 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:58,461 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:59:58,461 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:59:58,589 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:59,605 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:59:59,643 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:00,616 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:01,609 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:02,237 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:00:02,498 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:00:02,498 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:00:02,498 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:00:02,617 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:03,238 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:03,238 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:00:03,614 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:04,633 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:04,644 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:05,239 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:05,611 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:06,611 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:07,617 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:08,628 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:09,241 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:09,647 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:09,649 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:10,619 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:11,619 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:12,638 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:13,006 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:00:13,473 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:00:13,473 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:00:13,630 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:14,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:15,243 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:15,278 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:15,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:16,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:17,244 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:17,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:18,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:19,621 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:20,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:20,630 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:21,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:22,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:23,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:24,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:25,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:25,630 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:26,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:27,622 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:28,466 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:00:28,466 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:00:28,667 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:29,249 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:29,659 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:30,007 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:00:30,007 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 with policy now |
| 2025-01-15 22:00:30,032 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:00:30,032 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:00:30,033 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:00:30,033 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:00:30,249 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:00:30,250 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 |
| 2025-01-15 22:00:30,250 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 22:00:30,263 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/858lh38v-media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 |
| 2025-01-15 22:00:30,537 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/pt3wspel-media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 |
| 2025-01-15 22:00:30,675 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:31,064 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:31,250 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:31,670 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:32,669 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:33,251 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:33,737 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:34,713 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:35,252 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:35,716 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:36,660 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:36,711 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:37,720 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:38,719 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:39,253 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:39,712 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:40,722 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:41,673 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:41,741 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:42,720 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:43,007 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:00:43,473 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:00:43,474 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:00:43,741 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:44,737 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:45,256 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:00:45,532 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/ia05v5to-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:00:45,555 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:00:45,555 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:00:45,555 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:00:45,744 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:46,256 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:00:46,732 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:46,953 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:47,257 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:47,734 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:48,744 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:49,747 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:50,777 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:51,258 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:51,768 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:52,187 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:52,748 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:53,767 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:54,769 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:55,781 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:56,783 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:57,670 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:00:57,802 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:58,454 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:00:58,455 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:00:58,759 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:00:59,262 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:00:59,759 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:00,759 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:01,263 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:01:01,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:02,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:03,648 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:03,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:04,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:05,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:06,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:07,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:08,648 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:08,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:09,762 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:10,761 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:11,762 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:12,762 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:13,007 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:01:13,454 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:01:13,454 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:01:13,648 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:13,762 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:14,762 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:15,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:16,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:17,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:18,649 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:18,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:19,763 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:20,764 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:21,764 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:22,764 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:23,649 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:23,764 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:24,764 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:25,813 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:26,274 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 |
| 2025-01-15 22:01:26,274 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 22:01:26,287 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:01:26,288 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 with policy now |
| 2025-01-15 22:01:26,290 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:01:26,291 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:01:26,291 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:01:26,291 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:01:26,545 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/7disiecn-media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 |
| 2025-01-15 22:01:26,798 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:26,836 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/xc8psx8a-media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 |
| 2025-01-15 22:01:27,274 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:01:27,274 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:01:27,813 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:28,487 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:01:28,488 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:01:28,654 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:28,808 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:29,275 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:01:29,826 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:30,815 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:31,805 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:32,810 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:33,277 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:01:33,855 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:34,653 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:34,829 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:35,278 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:01:35,810 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:36,840 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:37,865 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:38,845 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:39,685 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:39,845 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:40,858 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:41,836 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:42,848 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:43,008 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:01:43,492 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:01:43,492 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:01:43,843 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:44,860 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:45,684 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:45,867 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:46,886 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:47,900 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:48,864 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:49,926 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:50,700 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:50,886 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:51,890 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:52,897 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:53,897 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:54,905 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:55,955 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:56,694 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:01:56,909 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:57,915 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:58,478 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:01:58,479 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:01:58,907 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:01:59,928 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:00,944 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:01,709 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:01,982 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:02,985 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:03,949 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:04,981 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:05,959 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:06,716 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:06,974 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:07,970 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:08,969 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:09,979 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:10,981 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:11,988 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:12,694 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:12,994 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:13,008 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:02:13,480 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:02:13,480 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:02:14,007 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:15,012 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:15,295 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:02:15,674 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:02:15,674 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:02:15,674 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:02:16,008 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:16,295 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:02:17,044 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:17,296 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:18,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:18,129 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:19,023 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:20,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:21,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:21,297 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:22,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:23,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:23,539 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:24,005 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:25,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:26,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:27,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:28,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:28,469 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:02:28,469 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:02:28,615 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:29,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:30,006 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:31,007 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:32,008 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:33,008 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:33,692 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:34,031 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:35,071 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:35,305 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:35,410 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:02:35,410 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 with policy now |
| 2025-01-15 22:02:35,423 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:02:35,423 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:02:35,423 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:02:35,424 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:02:35,646 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/4yu7df0h-media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 |
| 2025-01-15 22:02:36,063 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:36,306 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:02:36,306 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 |
| 2025-01-15 22:02:36,306 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 22:02:36,599 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/t1zmllnw-media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 |
| 2025-01-15 22:02:37,044 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:37,306 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:38,080 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:38,731 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:39,059 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:40,076 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:41,061 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:41,308 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:42,069 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:43,009 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:02:43,089 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:43,507 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:02:43,508 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:02:44,099 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:44,730 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:45,113 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:45,310 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:46,115 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:47,119 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:48,115 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:49,135 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:49,738 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:50,127 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:51,153 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:52,146 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:53,151 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:54,157 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:54,741 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:02:55,171 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:56,148 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:57,153 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:58,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:58,315 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:02:58,482 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:02:58,483 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:02:58,725 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:02:58,725 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:02:58,725 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:02:59,185 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:02:59,316 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:02:59,316 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:02:59,316 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:03:00,173 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:00,401 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:01,164 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:01,317 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:02,188 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:03,191 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:04,190 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:05,173 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:05,318 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:05,734 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:06,203 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:07,205 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:08,201 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:09,207 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:10,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:11,134 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:11,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:11,321 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:12,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:13,009 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:03:13,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:13,478 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:03:13,478 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:03:14,194 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:15,195 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:16,195 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:16,789 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:17,195 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:18,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:19,195 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:20,195 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:21,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:22,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:22,723 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:23,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:24,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:25,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:26,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:27,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:27,723 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:28,196 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:28,478 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:03:28,479 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:03:29,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:30,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:31,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:32,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:32,723 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:33,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:34,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:35,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:36,197 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:37,212 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:37,725 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:38,199 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:38,518 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:03:38,518 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 with policy now |
| 2025-01-15 22:03:38,526 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:03:38,526 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:03:38,526 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:03:38,527 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:03:38,777 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/n4px6l3n-media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 |
| 2025-01-15 22:03:39,210 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:39,333 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:39,333 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:03:39,334 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 |
| 2025-01-15 22:03:39,334 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 22:03:39,649 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/szqdmjnl-media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 |
| 2025-01-15 22:03:40,215 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:41,221 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:41,334 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:42,244 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:43,009 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:03:43,011 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:43,228 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:43,532 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:03:43,532 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:03:44,260 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:45,241 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:45,336 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:46,249 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:47,261 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:47,337 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:48,263 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:48,729 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:49,249 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:50,255 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:51,341 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:52,275 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:53,273 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:53,339 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:03:53,787 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:03:53,791 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:03:53,791 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:03:53,792 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:03:54,283 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:54,340 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:03:54,340 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:03:55,297 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:55,340 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:56,326 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:57,278 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:57,341 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:58,279 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:58,517 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:03:58,518 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:03:59,294 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:03:59,342 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:03:59,768 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:00,292 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:01,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:02,344 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:03,329 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:04,326 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:04,779 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:05,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:06,313 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:07,348 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:08,340 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:09,388 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:09,800 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:10,316 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:11,315 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:11,347 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:12,316 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:13,010 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:04:13,316 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:13,492 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:04:13,492 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:04:14,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:15,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:15,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:16,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:17,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:18,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:19,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:20,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:21,317 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:21,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:22,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:23,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:24,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:25,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:26,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:26,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:27,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:28,318 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:28,492 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:04:28,492 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:04:29,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:30,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:31,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:31,760 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:32,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:33,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:34,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:35,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:36,319 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:37,352 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:37,439 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:38,339 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:04:38,339 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 with policy now |
| 2025-01-15 22:04:38,359 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 |
| 2025-01-15 22:04:38,359 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval |
| 2025-01-15 22:04:38,370 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:04:38,377 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:04:38,377 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:38,377 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:38,379 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:38,606 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/tje53cgu-media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 |
| 2025-01-15 22:04:38,908 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/s9jrxm23-media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 |
| 2025-01-15 22:04:39,357 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:39,360 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:39,360 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:04:40,367 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:41,361 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:41,368 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:42,360 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:43,010 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:04:43,011 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:43,361 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:43,403 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:43,499 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:04:43,499 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:04:44,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:45,389 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:46,390 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:47,363 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:47,411 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:48,379 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:48,625 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:49,409 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:50,402 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:51,413 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:52,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:53,439 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:53,794 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:54,366 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:04:54,437 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:54,642 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/gms0en9u-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:04:54,864 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:04:54,864 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:54,864 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:55,367 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:04:55,367 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:55,367 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:04:55,446 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:55,970 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-15 22:04:55,971 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-15 22:04:56,367 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json |
| 2025-01-15 22:04:56,367 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media |
| 2025-01-15 22:04:56,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table |
| 2025-01-15 22:04:56,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval |
| 2025-01-15 22:04:56,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:56,449 INFO wandb-upload_3:2470873 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpnbdac6qz |
| 2025-01-15 22:04:56,476 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:04:56,568 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/5fe6wu6w-media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json |
| 2025-01-15 22:04:56,836 INFO SenderThread:2470873 [sender.py:send_request_log_artifact():1525] logged artifact run-roatk2bk-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0NDEwODI1NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzOTE1MDQ5NA==', 'latestArtifact': None}} |
| 2025-01-15 22:04:56,836 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:04:56,836 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json with policy now |
| 2025-01-15 22:04:56,836 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:04:56,836 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,837 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,837 DEBUG SenderThread:2470873 [sender.py:send():379] send: telemetry |
| 2025-01-15 22:04:56,837 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,838 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,838 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,838 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,839 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,840 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,841 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,841 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,841 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,842 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,842 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,842 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,842 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,843 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,843 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,843 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,844 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,844 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,844 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,845 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,845 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:56,845 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:56,875 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-15 22:04:56,875 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-15 22:04:56,887 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 22:04:57,140 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/ve85kqna-media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json |
| 2025-01-15 22:04:57,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:04:57,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:04:57,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json |
| 2025-01-15 22:04:57,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval |
| 2025-01-15 22:04:57,389 INFO wandb-upload_3:2470873 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpuhxzzwc7 |
| 2025-01-15 22:04:57,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:57,563 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/44kyffge-media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json |
| 2025-01-15 22:04:57,762 INFO SenderThread:2470873 [sender.py:send_request_log_artifact():1525] logged artifact run-roatk2bk-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0NDEwODI2Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzOTE1MDQ5Ng==', 'latestArtifact': None}} |
| 2025-01-15 22:04:57,762 DEBUG SenderThread:2470873 [sender.py:send():379] send: files |
| 2025-01-15 22:04:57,762 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json with policy now |
| 2025-01-15 22:04:57,763 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:04:57,763 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:04:57,763 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:04:58,031 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/ydokvtt1-media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json |
| 2025-01-15 22:04:58,368 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:04:58,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:04:58,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:04:58,496 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:04:58,798 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:04:59,369 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:04:59,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:00,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:01,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:02,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:03,406 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:03,802 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:05:04,372 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/config.yaml |
| 2025-01-15 22:05:04,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:05,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:06,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:07,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:08,407 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:09,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:09,799 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:05:10,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:11,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:12,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:13,011 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:05:13,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:13,496 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 22:05:13,496 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 22:05:14,408 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:14,841 DEBUG SenderThread:2470873 [sender.py:send():379] send: exit |
| 2025-01-15 22:05:14,841 INFO SenderThread:2470873 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-15 22:05:14,842 INFO SenderThread:2470873 [sender.py:send_exit():588] handling runtime: 1201 |
| 2025-01-15 22:05:14,842 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:05:14,842 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:05:14,842 INFO SenderThread:2470873 [sender.py:send_exit():594] send defer |
| 2025-01-15 22:05:14,842 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:14,842 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-15 22:05:14,843 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:14,843 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-15 22:05:14,843 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-15 22:05:14,843 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:14,843 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-15 22:05:14,843 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:14,843 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-15 22:05:14,843 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-15 22:05:14,843 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:14,843 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-15 22:05:14,843 INFO HandlerThread:2470873 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-15 22:05:14,843 DEBUG SystemMonitor:2470873 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-15 22:05:14,844 INFO HandlerThread:2470873 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-15 22:05:14,844 DEBUG SystemMonitor:2470873 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-15 22:05:14,844 INFO HandlerThread:2470873 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-15 22:05:14,915 INFO HandlerThread:2470873 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-15 22:05:14,915 INFO HandlerThread:2470873 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-15 22:05:14,915 INFO HandlerThread:2470873 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-15 22:05:14,915 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:14,915 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-15 22:05:14,916 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-15 22:05:14,916 DEBUG SenderThread:2470873 [sender.py:send():379] send: stats |
| 2025-01-15 22:05:14,916 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:14,916 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-15 22:05:14,916 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:14,916 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-15 22:05:14,916 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-15 22:05:14,916 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:14,916 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-15 22:05:15,378 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:05:17,379 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:05:21,872 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 22:05:21,873 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:05:21,873 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-15 22:05:21,874 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:21,874 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-15 22:05:21,874 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-15 22:05:21,874 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 22:05:21,874 DEBUG SenderThread:2470873 [sender.py:send():379] send: history |
| 2025-01-15 22:05:21,874 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 22:05:21,874 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:21,875 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-15 22:05:21,875 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:05:21,875 DEBUG SenderThread:2470873 [sender.py:send():379] send: summary |
| 2025-01-15 22:05:21,879 INFO SenderThread:2470873 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 22:05:21,879 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:21,879 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-15 22:05:21,879 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-15 22:05:21,879 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:21,880 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-15 22:05:21,880 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:21,880 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-15 22:05:21,880 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-15 22:05:21,880 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 22:05:21,880 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:21,880 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-15 22:05:21,880 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:21,880 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-15 22:05:22,381 INFO Thread-12 :2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:05:22,843 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 22:05:23,031 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-15 22:05:23,031 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 22:05:23,031 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:23,031 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-15 22:05:23,031 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:23,032 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-15 22:05:23,032 INFO SenderThread:2470873 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-15 22:05:23,035 INFO SenderThread:2470873 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-15 22:05:23,036 INFO SenderThread:2470873 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-15 22:05:23,049 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-15 22:05:23,049 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:23,049 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-15 22:05:23,050 DEBUG SenderThread:2470873 [sender.py:send():379] send: artifact |
| 2025-01-15 22:05:23,221 INFO SenderThread:2470873 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ0NDA3MjMyOQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ0NDA3MjMyOQ==', 'versionIndex': 440}}} |
| 2025-01-15 22:05:23,221 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:23,221 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-15 22:05:23,221 INFO SenderThread:2470873 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-15 22:05:23,382 INFO SenderThread:2470873 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:05:23,383 INFO SenderThread:2470873 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files |
| 2025-01-15 22:05:23,383 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/config.yaml config.yaml |
| 2025-01-15 22:05:23,383 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json wandb-summary.json |
| 2025-01-15 22:05:23,383 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/requirements.txt requirements.txt |
| 2025-01-15 22:05:23,386 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-15 22:05:23,386 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:05:23,386 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 22:05:23,386 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log output.log |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/diff.patch diff.patch |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 media/videos/eval/3_eval_3_89821fb09b372630838b.mp4 |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 media/videos/eval/4_eval_4_9fb52356d0a213033cfe.mp4 |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 media/videos/eval/10_eval_10_6e43fe2801f3a905e582.mp4 |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 media/videos/eval/5_eval_5_a69fcdfd49e2066116b6.mp4 |
| 2025-01-15 22:05:23,388 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 media/videos/eval/0_eval_0_2eb6cf19687a4fabfe3c.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 media/videos/eval/18_eval_18_c8c84019c8dc8ba0e4ea.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 media/videos/eval/15_eval_15_76fa4e81e9e73fef9cf0.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 media/videos/eval/19_eval_19_3f737e19fed74dd298b2.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 media/videos/eval/17_eval_17_2f96b121fd6f99c60674.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 media/videos/eval/12_eval_12_241be90b3f1c54884e79.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 media/videos/eval/13_eval_13_cb3449f02cfc7cf55b8d.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 media/videos/eval/1_eval_1_b5d7870b7b04a6ef485f.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 media/videos/eval/2_eval_2_f98c31cd2e05324cffd8.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 media/videos/eval/6_eval_6_ae418a6c99a51b4201d6.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 media/videos/eval/14_eval_14_074bdfc80a4a9d765f98.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 media/videos/eval/16_eval_16_5019757f76965e10782b.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 media/videos/eval/7_eval_7_f02cea2dbceefebb3b97.mp4 |
| 2025-01-15 22:05:23,389 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 media/videos/eval/9_eval_9_19795df8831a10c10837.mp4 |
| 2025-01-15 22:05:23,390 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 media/videos/eval/8_eval_8_fb8f14c1c353cc80983a.mp4 |
| 2025-01-15 22:05:23,390 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 media/videos/eval/11_eval_11_4263c7dc48b07a1c552c.mp4 |
| 2025-01-15 22:05:23,390 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json media/table/eval/total_success_rate_ci_21_ee38b3327b46c4981cc4.table.json |
| 2025-01-15 22:05:23,390 INFO SenderThread:2470873 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json media/table/eval/success_rate_ci_20_0f278608fd5017506fe4.table.json |
| 2025-01-15 22:05:23,390 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-15 22:05:23,392 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:23,393 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-15 22:05:23,393 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:23,393 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-15 22:05:23,393 INFO SenderThread:2470873 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-15 22:05:23,569 INFO wandb-upload_2:2470873 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/wandb-summary.json |
| 2025-01-15 22:05:23,575 INFO wandb-upload_3:2470873 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/config.yaml |
| 2025-01-15 22:05:23,597 INFO wandb-upload_1:2470873 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/requirements.txt |
| 2025-01-15 22:05:23,660 INFO wandb-upload_4:2470873 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/files/output.log |
| 2025-01-15 22:05:23,707 INFO wandb-upload_0:2470873 [upload_job.py:push():130] Uploaded file /tmp/tmpt0febobwwandb/niuudu4q-events.out.tfevents.1736995513.leonmkim-ROG-Strix-G15CS-G15CS.2470707.0 |
| 2025-01-15 22:05:23,843 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 22:05:23,843 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 22:05:23,908 INFO Thread-11 (_thread_body):2470873 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-15 22:05:23,908 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:23,908 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-15 22:05:23,908 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:23,908 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-15 22:05:23,908 INFO SenderThread:2470873 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-15 22:05:23,916 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-15 22:05:23,916 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:23,916 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-15 22:05:23,916 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:23,916 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-15 22:05:23,916 INFO SenderThread:2470873 [file_stream.py:finish():601] file stream finish called |
| 2025-01-15 22:05:24,132 INFO SenderThread:2470873 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-15 22:05:24,132 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-15 22:05:24,133 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:24,133 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-15 22:05:24,133 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:24,133 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-15 22:05:24,133 INFO SenderThread:2470873 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-15 22:05:24,133 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 22:05:24,133 INFO HandlerThread:2470873 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-15 22:05:24,133 DEBUG SenderThread:2470873 [sender.py:send():379] send: final |
| 2025-01-15 22:05:24,133 DEBUG SenderThread:2470873 [sender.py:send():379] send: footer |
| 2025-01-15 22:05:24,133 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 22:05:24,133 INFO SenderThread:2470873 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-15 22:05:24,134 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 22:05:24,134 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-15 22:05:24,134 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 22:05:24,134 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 22:05:24,135 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: server_info |
| 2025-01-15 22:05:24,136 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-15 22:05:24,137 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-15 22:05:24,138 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 22:05:24,187 DEBUG SenderThread:2470873 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 22:05:24,188 INFO MainThread:2470873 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-15 22:05:24,188 INFO MainThread:2470873 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-15 22:05:24,189 INFO MainThread:2470873 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-15 22:05:24,189 DEBUG HandlerThread:2470873 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-15 22:05:24,189 INFO HandlerThread:2470873 [handler.py:finish():882] shutting down handler |
| 2025-01-15 22:05:25,135 INFO WriterThread:2470873 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/roatk2bk/wandb/run-20250115_214512-roatk2bk/run-roatk2bk.wandb |
| 2025-01-15 22:05:25,188 INFO SenderThread:2470873 [sender.py:finish():1615] shutting down sender |
| 2025-01-15 22:05:25,188 INFO SenderThread:2470873 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-15 22:05:25,188 INFO SenderThread:2470873 [file_pusher.py:join():175] waiting for file pusher |
|
|
