| 2025-01-15 21:25:12,918 INFO StreamThr :2467459 [internal.py:wandb_internal():85] W&B internal server running at pid: 2467459, started at: 2025-01-15 21:25:12.918214 |
| 2025-01-15 21:25:12,919 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status |
| 2025-01-15 21:25:12,929 INFO WriterThread:2467459 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/run-vfeb54wi.wandb |
| 2025-01-15 21:25:12,929 DEBUG SenderThread:2467459 [sender.py:send():379] send: header |
| 2025-01-15 21:25:12,938 DEBUG SenderThread:2467459 [sender.py:send():379] send: run |
| 2025-01-15 21:25:13,150 INFO SenderThread:2467459 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files |
| 2025-01-15 21:25:13,150 INFO SenderThread:2467459 [sender.py:_start_run_threads():1188] run started: vfeb54wi with start time 1736994312.918604 |
| 2025-01-15 21:25:13,164 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-15 21:25:13,164 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: check_version |
| 2025-01-15 21:25:13,218 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-15 21:25:13,223 DEBUG HandlerThread:2467459 [system_info.py:__init__():26] System info init |
| 2025-01-15 21:25:13,223 DEBUG HandlerThread:2467459 [system_info.py:__init__():41] System info init done |
| 2025-01-15 21:25:13,223 INFO HandlerThread:2467459 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-15 21:25:13,223 INFO SystemMonitor:2467459 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-15 21:25:13,223 INFO HandlerThread:2467459 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-15 21:25:13,224 INFO SystemMonitor:2467459 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-15 21:25:13,224 INFO SystemMonitor:2467459 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-15 21:25:13,225 INFO SystemMonitor:2467459 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-15 21:25:13,225 INFO SystemMonitor:2467459 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-15 21:25:13,225 INFO SystemMonitor:2467459 [interfaces.py:start():188] Started network monitoring |
| 2025-01-15 21:25:13,236 DEBUG HandlerThread:2467459 [system_info.py:probe():152] Probing system |
| 2025-01-15 21:25:13,238 DEBUG HandlerThread:2467459 [system_info.py:_probe_git():137] Probing git |
| 2025-01-15 21:25:13,242 DEBUG HandlerThread:2467459 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-15 21:25:13,243 DEBUG HandlerThread:2467459 [system_info.py:probe():200] Probing system done |
| 2025-01-15 21:25:13,243 DEBUG HandlerThread:2467459 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-16T02:25:13.236976', 'startedAt': '2025-01-16T02:25:12.910503', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3937.49975, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.962, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 302.54129791259766}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-15 21:25:13,243 INFO HandlerThread:2467459 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-15 21:25:13,243 INFO HandlerThread:2467459 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-15 21:25:13,243 DEBUG HandlerThread:2467459 [system_info.py:_save_code():44] Saving code |
| 2025-01-15 21:25:13,244 DEBUG HandlerThread:2467459 [system_info.py:_save_code():67] Saving code done |
| 2025-01-15 21:25:13,244 DEBUG HandlerThread:2467459 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-15 21:25:13,340 DEBUG HandlerThread:2467459 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-15 21:25:13,341 INFO HandlerThread:2467459 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-15 21:25:13,344 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:25:13,344 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-15 21:25:13,344 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-15 21:25:13,344 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-15 21:25:13,345 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-15 21:25:13,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-15 21:25:13,583 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-15 21:25:13,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:25:13,583 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:25:13,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:13,599 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/fdj573ij-wandb-metadata.json |
| 2025-01-15 21:25:13,609 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/jmoc2a32-code/FISH/eval_robot.py |
| 2025-01-15 21:25:13,620 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/346yqiic-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 21:25:13,623 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/4asr6hxf-diff.patch |
| 2025-01-15 21:25:13,629 INFO HandlerThread:2467459 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/tb" |
| save: true |
| } |
| control { |
| relay_id: "83dc957e0c7848d8aed4259b6fd3c7e4" |
| } |
| _info { |
| stream_id: "vfeb54wi" |
| } |
|
|
| 2025-01-15 21:25:13,629 DEBUG HandlerThread:2467459 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-15 21:25:13,666 DEBUG SenderThread:2467459 [sender.py:send():379] send: telemetry |
| 2025-01-15 21:25:13,697 DEBUG SenderThread:2467459 [sender.py:send():379] send: tbrecord |
| 2025-01-15 21:25:13,698 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:25:13,698 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 with policy live |
| 2025-01-15 21:25:13,891 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/6y9bdqf3-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:25:14,151 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/requirements.txt |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/code/FISH/eval_robot.py |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/diff.patch |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-metadata.json |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/code/FISH |
| 2025-01-15 21:25:14,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/code |
| 2025-01-15 21:25:14,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:15,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:16,151 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:16,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:17,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:18,431 DEBUG SenderThread:2467459 [sender.py:send():379] send: config |
| 2025-01-15 21:25:18,431 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:18,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:19,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:20,152 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:20,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:21,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:22,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:23,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:23,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:24,153 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:24,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:25,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:26,154 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:26,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:27,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:28,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:25:28,583 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:25:28,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:28,727 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:29,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:30,155 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:25:30,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:31,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:32,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:33,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:34,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:34,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:35,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:36,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:37,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:38,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:39,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:39,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:40,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:41,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:42,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:43,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:25:43,583 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:25:43,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:44,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:44,719 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:45,159 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/config.yaml |
| 2025-01-15 21:25:45,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:46,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:47,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:48,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:49,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:50,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:50,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:51,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:52,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:53,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:54,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:55,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:56,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:25:56,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:57,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:58,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:25:58,583 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:25:58,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:25:59,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:00,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:01,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:01,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:02,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:03,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:04,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:05,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:06,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:07,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:07,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:08,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:09,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:10,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:11,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:12,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:12,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:13,225 DEBUG SystemMonitor:2467459 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-15 21:26:13,226 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:26:13,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:26:13,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:26:13,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:14,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:15,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:16,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:17,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:17,687 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:18,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:19,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:20,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:21,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:22,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:23,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:23,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:24,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:25,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:26,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:27,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:28,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:26:28,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:26:28,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:28,650 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:29,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:30,172 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:26:30,595 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:31,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:32,172 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:26:32,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:33,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:34,173 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:26:34,191 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:34,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:35,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:36,173 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:26:36,596 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:37,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:38,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:39,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:39,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:40,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:41,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:42,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:43,227 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:26:43,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:26:43,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:26:43,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:44,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:44,688 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:45,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:46,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:47,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:48,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:49,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:50,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:50,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:51,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:52,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:53,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:54,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:55,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:26:55,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:56,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:57,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:58,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:26:58,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:26:58,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:26:58,973 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:26:58,973 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 with policy now |
| 2025-01-15 21:26:58,973 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:26:58,974 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:26:58,974 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:26:58,976 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:26:59,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 |
| 2025-01-15 21:26:59,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:26:59,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:26:59,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media |
| 2025-01-15 21:26:59,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos |
| 2025-01-15 21:26:59,209 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/kjtcrf0m-media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 |
| 2025-01-15 21:26:59,476 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/s0tmh9b8-media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 |
| 2025-01-15 21:26:59,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:00,180 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:00,602 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:00,980 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:01,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:02,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:03,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:04,182 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:04,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:05,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:06,182 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:06,333 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:06,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:07,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:08,183 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:08,602 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:09,602 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:10,602 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:11,588 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:11,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:12,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:13,227 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:27:13,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:27:13,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:27:13,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:14,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:15,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:16,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:16,736 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:17,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:18,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:19,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:20,603 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:21,604 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:22,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:22,604 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:23,604 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:24,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:25,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:26,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:27,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:27,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:28,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:27:28,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:27:28,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:29,190 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:27:29,450 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/twg1wjqp-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:27:29,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:30,191 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:30,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:31,605 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:32,192 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:32,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:32,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:33,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:34,192 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:34,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:35,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:36,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:37,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:38,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:38,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:39,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:40,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:41,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:42,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:43,228 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:27:43,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:27:43,584 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:27:43,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:43,680 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:44,607 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:45,607 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:46,607 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:47,607 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:48,608 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:49,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:49,608 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:50,198 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:50,608 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:51,609 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:52,609 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:53,609 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:54,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:27:54,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:55,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:56,200 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:56,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:57,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:58,200 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:27:58,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:27:58,585 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:27:58,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:59,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:27:59,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:00,201 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:00,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:01,080 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:28:01,080 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:28:01,080 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 with policy now |
| 2025-01-15 21:28:01,081 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:28:01,081 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:28:01,094 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:28:01,202 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:28:01,202 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 |
| 2025-01-15 21:28:01,202 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:28:01,306 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/rfbzb3bw-media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 |
| 2025-01-15 21:28:01,573 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/0x6crfkr-media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 |
| 2025-01-15 21:28:01,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:02,202 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:02,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:03,611 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:04,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:04,702 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:05,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:06,204 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:06,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:07,611 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:08,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:09,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:10,205 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:10,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:10,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:11,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:12,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:13,228 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:28:13,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:28:13,586 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:28:13,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:14,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:15,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:15,709 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:16,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:17,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:18,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:19,614 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:20,615 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:21,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:21,615 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:22,615 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:23,209 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:28:23,616 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:24,141 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:28:24,141 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:28:24,142 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:28:24,210 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:24,210 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:28:24,616 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:25,616 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:26,210 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:26,616 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:26,846 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:27,616 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:28,211 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:28,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:28:28,585 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:28:28,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:29,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:30,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:31,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:32,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:32,710 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:33,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:34,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:35,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:36,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:37,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:38,501 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:38,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:39,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:40,215 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:40,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:41,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:42,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:43,229 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:28:43,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:28:43,585 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:28:43,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:43,651 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:44,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:45,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:46,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:47,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:48,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:49,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:49,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:50,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:51,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:52,619 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:53,619 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:54,604 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:28:54,623 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:55,273 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:28:55,273 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 with policy now |
| 2025-01-15 21:28:55,276 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:28:55,277 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:28:55,277 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:28:55,283 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:28:55,490 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/h8kv98eg-media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 |
| 2025-01-15 21:28:55,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:56,220 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:56,221 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:28:56,221 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 |
| 2025-01-15 21:28:56,221 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:28:56,513 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/nframv28-media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 |
| 2025-01-15 21:28:56,626 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:57,221 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:28:57,627 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:58,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:28:58,588 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:28:58,633 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:59,624 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:28:59,961 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:00,222 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:00,626 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:01,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:02,624 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:03,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:04,224 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:04,632 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:05,592 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:05,630 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:06,628 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:07,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:08,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:08,978 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:29:08,978 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:29:08,978 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:29:09,227 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:29:09,227 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:29:09,488 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/t9k3stc7-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:29:09,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:10,227 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:10,638 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:10,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:11,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:12,228 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:12,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:13,229 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:29:13,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:29:13,586 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:29:13,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:14,229 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:14,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:15,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:16,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:16,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:17,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:18,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:19,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:20,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:21,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:22,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:22,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:23,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:24,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:25,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:26,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:27,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:27,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:28,586 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:29:28,587 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:29:28,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:29,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:30,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:31,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:32,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:32,738 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:33,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:34,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:35,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:36,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:37,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:38,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:38,636 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:39,637 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:40,238 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:40,647 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:40,966 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:29:40,966 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 with policy now |
| 2025-01-15 21:29:40,970 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:29:40,971 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:29:40,971 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:29:40,971 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:29:41,207 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/qxd14u76-media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 |
| 2025-01-15 21:29:41,238 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:29:41,238 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 |
| 2025-01-15 21:29:41,239 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:29:41,494 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/v035hc8i-media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 |
| 2025-01-15 21:29:41,648 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:42,239 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:42,653 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:43,229 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:29:43,239 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:43,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:29:43,588 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:29:43,648 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:43,720 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:44,650 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:45,668 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:46,240 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:46,657 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:47,654 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:48,241 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:48,660 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:49,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:49,661 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:50,660 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:51,661 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:52,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:53,663 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:54,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:55,032 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:29:55,033 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:29:55,033 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:29:55,034 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:29:55,243 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:29:55,243 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:29:55,667 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:56,243 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:29:56,672 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:57,669 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:58,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:29:58,587 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:29:58,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:29:59,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:00,245 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:00,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:00,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:01,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:02,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:03,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:04,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:05,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:06,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:06,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:07,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:08,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:09,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:10,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:11,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:11,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:12,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:13,230 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:30:13,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:30:13,587 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:30:13,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:14,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:15,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:16,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:16,738 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:17,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:18,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:19,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:20,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:21,667 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:22,598 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:22,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:23,667 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:24,667 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:25,675 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:26,242 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:30:26,242 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 with policy now |
| 2025-01-15 21:30:26,245 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:30:26,246 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:30:26,246 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:30:26,246 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:30:26,254 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:26,255 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:30:26,255 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 |
| 2025-01-15 21:30:26,255 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:30:26,459 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/wdw28quw-media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 |
| 2025-01-15 21:30:26,674 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:26,725 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/a1s92lgd-media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 |
| 2025-01-15 21:30:27,683 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:28,255 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:28,268 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:28,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:30:28,589 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:30:28,684 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:29,682 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:30,684 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:31,687 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:32,256 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:32,685 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:33,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:33,697 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:34,257 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:34,686 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:35,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:36,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:37,693 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:38,599 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:38,694 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:39,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:40,088 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:30:40,088 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:30:40,090 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:30:40,259 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:30:40,259 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:30:40,509 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/bziwa5lu-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:30:40,701 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:41,697 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:42,260 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:42,700 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:43,230 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:30:43,609 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:30:43,610 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:30:43,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:43,694 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:44,260 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:44,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:45,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:46,261 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:46,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:47,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:48,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:49,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:49,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:50,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:51,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:52,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:53,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:54,264 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:30:54,601 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:54,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:55,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:56,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:57,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:58,589 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:30:58,589 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:30:58,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:30:59,669 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:30:59,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:00,697 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:01,072 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:31:01,072 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 with policy now |
| 2025-01-15 21:31:01,079 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:31:01,079 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:31:01,079 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:31:01,080 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:31:01,267 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:31:01,267 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 |
| 2025-01-15 21:31:01,267 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:31:01,278 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ckcq6zx5-media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 |
| 2025-01-15 21:31:01,538 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/1osnfgsm-media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 |
| 2025-01-15 21:31:01,696 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:02,267 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:02,697 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:03,709 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:04,268 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:04,710 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:04,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:05,701 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:06,268 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:06,702 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:07,698 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:08,269 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:08,707 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:09,708 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:10,270 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:10,610 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:10,703 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:11,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:12,709 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:13,231 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:31:13,602 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:31:13,602 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:31:13,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:14,721 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:15,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:15,753 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:16,727 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:17,135 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:31:17,135 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:31:17,135 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:31:17,272 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:31:17,273 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:31:17,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:18,273 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:18,722 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:19,735 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:20,273 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:20,732 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:20,785 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:21,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:22,274 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:22,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:23,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:24,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:25,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:26,608 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:26,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:27,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:28,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:31:28,591 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:31:28,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:29,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:30,724 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:31,691 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:31,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:32,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:33,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:34,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:35,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:36,728 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:37,053 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:37,400 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:31:37,400 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 with policy now |
| 2025-01-15 21:31:37,423 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:31:37,423 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:31:37,423 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:31:37,424 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:31:37,632 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/9oj5qj44-media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 |
| 2025-01-15 21:31:37,735 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:38,280 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:38,280 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:31:38,280 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 |
| 2025-01-15 21:31:38,280 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:31:38,548 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ll3s455q-media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 |
| 2025-01-15 21:31:38,728 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:39,731 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:40,281 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:40,734 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:41,739 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:42,281 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:42,740 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:43,041 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:43,231 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:31:43,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:31:43,594 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:31:43,751 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:44,752 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:45,738 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:46,284 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:46,753 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:47,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:48,620 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:48,753 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:49,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:50,769 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:51,179 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:31:51,179 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:31:51,180 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:31:51,285 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:31:51,286 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:31:51,539 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/nwvjd0p0-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:31:51,755 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:52,286 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:52,760 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:53,764 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:54,276 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:54,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:55,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:56,287 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:31:56,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:57,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:58,591 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:31:58,591 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:31:58,756 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:31:59,720 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:31:59,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:00,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:01,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:02,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:03,757 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:04,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:05,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:05,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:06,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:07,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:08,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:09,777 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:10,261 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:32:10,261 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 with policy now |
| 2025-01-15 21:32:10,271 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:32:10,271 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:32:10,271 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:32:10,272 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:32:10,292 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:10,292 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:32:10,293 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 |
| 2025-01-15 21:32:10,293 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:32:10,473 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/b50a1vd9-media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 |
| 2025-01-15 21:32:10,735 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/yar3cn8d-media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 |
| 2025-01-15 21:32:10,771 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:11,305 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:11,774 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:12,293 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:12,783 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:13,232 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:32:13,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:32:13,600 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:32:13,774 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:14,773 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:15,784 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:16,294 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:16,782 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:16,799 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:17,786 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:18,295 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:18,790 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:19,788 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:20,787 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:21,790 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:22,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:22,789 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:23,791 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:24,817 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:25,795 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:26,793 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:27,621 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:27,807 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:28,606 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:32:28,607 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:32:28,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:29,818 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:30,821 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:31,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:32,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:32,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:33,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:34,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:35,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:36,235 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:32:36,235 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:32:36,235 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:32:36,302 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:32:36,302 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:32:36,818 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:37,816 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:38,257 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:38,302 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:38,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:39,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:40,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:41,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:42,304 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:42,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:43,232 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:32:43,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:43,629 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:32:43,629 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:32:43,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:44,305 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:44,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:45,812 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:46,306 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:32:46,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:47,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:48,619 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:48,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:49,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:50,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:51,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:52,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:53,620 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:53,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:54,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:55,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:56,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:57,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:58,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:32:58,593 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:32:58,752 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:32:58,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:32:59,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:00,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:01,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:02,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:03,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:04,622 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:04,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:05,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:06,815 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:07,824 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:08,314 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:08,740 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:33:08,740 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 with policy now |
| 2025-01-15 21:33:08,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:33:08,758 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:33:08,758 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:33:08,759 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:33:08,823 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:09,052 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/qbg4vmhh-media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 |
| 2025-01-15 21:33:09,314 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:33:09,314 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 |
| 2025-01-15 21:33:09,314 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:33:09,612 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/lygq9vxg-media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 |
| 2025-01-15 21:33:09,803 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:09,829 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:10,315 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:10,826 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:11,846 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:12,843 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:13,233 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:33:13,621 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:33:13,622 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:33:13,827 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:14,316 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:14,835 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:15,389 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:15,825 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:16,317 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:16,828 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:17,837 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:18,837 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:19,834 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:20,640 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:20,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:21,844 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:22,291 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:33:22,291 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:33:22,292 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:33:22,319 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:33:22,319 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:22,319 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:33:22,571 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/h9u60pi3-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:33:22,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:23,842 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:24,320 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:24,862 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:25,686 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:25,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:26,321 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:26,837 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:27,837 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:28,322 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:28,593 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:33:28,594 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:33:28,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:29,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:30,765 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:30,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:31,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:32,838 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:33,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:34,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:35,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:36,113 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:36,325 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:36,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:37,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:38,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:39,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:40,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:41,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:41,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:42,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:43,233 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:33:43,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:33:43,594 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:33:43,839 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:44,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:45,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:46,689 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:46,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:47,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:48,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:49,840 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:50,841 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:51,841 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:52,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:52,841 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:53,853 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:54,313 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:33:54,313 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 with policy now |
| 2025-01-15 21:33:54,332 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:54,332 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 |
| 2025-01-15 21:33:54,332 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:33:54,335 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:33:54,336 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:33:54,336 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:33:54,336 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:33:54,544 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/mdztdjn2-media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 |
| 2025-01-15 21:33:54,811 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/iswk2gaz-media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 |
| 2025-01-15 21:33:54,848 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:55,332 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:33:55,854 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:56,333 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:33:56,857 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:57,871 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:57,999 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:33:58,612 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:33:58,612 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:33:58,858 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:33:59,869 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:00,334 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:00,877 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:01,878 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:02,335 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:02,880 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:03,025 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:03,887 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:04,889 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:05,880 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:06,336 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:34:06,337 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:06,346 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:34:06,346 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:34:06,346 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:34:06,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:07,337 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:34:07,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:08,338 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:08,679 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:08,913 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:09,899 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:10,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:11,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:12,341 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:12,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:13,234 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:34:13,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:34:13,597 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:34:13,741 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:13,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:14,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:15,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:16,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:17,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:18,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:19,650 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:19,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:20,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:21,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:22,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:23,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:24,650 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:24,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:25,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:26,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:27,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:28,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:34:28,597 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:34:28,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:29,758 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:29,893 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:30,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:31,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:32,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:33,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:34,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:35,651 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:35,894 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:36,895 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:37,930 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:38,351 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:38,412 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:34:38,412 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 with policy now |
| 2025-01-15 21:34:38,432 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:34:38,432 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:34:38,432 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:34:38,433 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:34:38,634 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/lf60jaaj-media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 |
| 2025-01-15 21:34:38,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:39,351 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:34:39,351 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 |
| 2025-01-15 21:34:39,352 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:34:39,630 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/qhot98oe-media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 |
| 2025-01-15 21:34:39,907 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:40,352 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:40,656 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:40,933 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:41,941 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:42,944 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:43,234 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:34:43,608 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:34:43,608 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:34:43,943 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:44,353 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:44,938 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:45,956 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:46,293 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:46,354 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:46,946 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:47,944 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:48,942 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:49,942 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:50,355 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:34:50,356 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:50,401 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:34:50,401 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:34:50,401 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:34:50,605 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ix0sfkor-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:34:50,954 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:51,356 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:34:51,930 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:51,946 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:52,356 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:52,961 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:53,958 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:54,965 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:55,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:56,358 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:34:56,977 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:57,045 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:34:57,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:58,621 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:34:58,621 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:34:58,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:34:59,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:00,361 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:00,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:01,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:02,077 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:02,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:03,966 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:04,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:05,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:06,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:07,663 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:07,967 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:08,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:09,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:10,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:11,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:12,663 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:12,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:13,235 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:35:13,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:35:13,598 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:35:13,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:14,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:15,968 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:16,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:17,693 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:17,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:18,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:19,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:20,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:21,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:22,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:23,663 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:23,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:24,969 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:25,971 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:26,371 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:27,026 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:27,203 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:35:27,204 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 with policy now |
| 2025-01-15 21:35:27,216 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:35:27,217 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:35:27,217 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:35:27,217 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:35:27,371 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:35:27,372 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 |
| 2025-01-15 21:35:27,372 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:35:27,429 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/kgx5r1rf-media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 |
| 2025-01-15 21:35:27,710 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/d4et035n-media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 |
| 2025-01-15 21:35:28,006 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:28,372 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:28,635 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:35:28,636 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:35:28,717 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:29,017 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:30,024 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:30,373 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:31,053 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:32,037 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:32,373 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:33,029 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:34,042 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:34,142 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:35,032 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:36,034 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:36,375 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:37,033 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:38,044 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:39,059 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:39,678 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:40,087 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:41,078 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:42,057 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:43,073 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:43,235 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:35:43,378 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:35:43,465 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:35:43,465 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:35:43,466 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:35:43,625 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:35:43,625 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:35:44,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:44,378 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:35:44,378 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:44,378 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:35:45,091 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:45,653 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:46,095 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:46,379 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:47,106 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:48,081 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:48,380 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:49,080 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:50,080 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:50,381 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:35:50,675 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:51,081 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:52,081 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:53,081 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:54,081 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:55,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:55,675 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:35:56,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:57,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:58,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:35:58,617 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:35:58,618 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:35:59,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:00,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:00,764 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:01,082 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:02,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:03,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:04,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:05,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:06,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:06,675 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:07,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:08,083 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:09,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:10,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:10,389 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:11,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:12,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:12,225 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:13,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:13,235 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:36:13,618 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:36:13,618 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:36:14,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:14,390 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:15,084 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:16,104 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:16,391 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:16,591 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:36:16,591 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 with policy now |
| 2025-01-15 21:36:16,600 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:36:16,601 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:36:16,601 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:36:16,601 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:36:16,804 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/i49yfhes-media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 |
| 2025-01-15 21:36:17,102 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:17,392 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:36:17,392 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 |
| 2025-01-15 21:36:17,392 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:36:17,663 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/xnhcm2ii-media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 |
| 2025-01-15 21:36:17,674 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:18,122 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:18,392 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:19,103 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:20,124 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:21,132 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:22,138 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:22,394 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:23,128 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:23,429 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:24,155 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:24,394 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:25,158 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:26,155 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:27,160 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:28,163 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:28,648 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:36:28,648 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:36:28,761 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:29,147 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:30,149 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:31,160 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:32,167 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:32,398 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:36:32,398 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:32,527 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:36:32,527 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:36:32,527 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:36:33,183 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:33,398 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:36:34,179 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:34,399 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:34,725 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:35,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:36,164 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:36,399 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:37,184 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:38,183 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:38,400 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:39,206 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:40,178 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:40,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:41,194 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:42,188 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:43,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:43,236 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:36:43,622 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:36:43,622 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:36:44,180 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:44,403 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:36:45,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:46,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:46,688 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:47,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:48,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:49,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:50,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:51,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:51,689 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:52,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:53,181 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:54,182 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:55,182 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:56,182 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:57,182 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:57,688 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:36:58,182 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:36:58,622 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:36:58,622 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:36:59,198 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:00,014 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:37:00,014 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 with policy now |
| 2025-01-15 21:37:00,025 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:37:00,025 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:37:00,025 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:37:00,026 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:37:00,190 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:00,230 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/q2rr96o6-media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 |
| 2025-01-15 21:37:00,409 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:00,409 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:37:00,409 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 |
| 2025-01-15 21:37:00,410 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:37:00,677 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/rl4f38nx-media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 |
| 2025-01-15 21:37:01,230 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:02,192 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:02,410 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:03,090 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:03,205 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:04,204 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:04,411 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:05,203 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:06,226 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:07,196 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:08,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:08,412 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:08,728 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:09,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:10,219 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:11,210 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:12,225 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:13,226 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:13,236 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:37:13,575 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:37:13,576 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:37:13,576 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:37:13,656 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:37:13,656 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:37:13,746 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:14,267 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:14,415 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:37:14,415 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:14,415 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:37:14,666 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/npemfisv-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:37:15,228 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:16,221 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:16,416 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:17,228 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:18,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:18,416 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:19,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:19,698 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:20,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:20,417 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:21,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:22,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:23,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:24,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:24,698 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:25,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:26,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:27,220 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:28,221 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:28,623 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:37:28,623 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:37:29,221 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:29,723 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:30,221 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:31,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:32,221 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:33,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:33,423 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:34,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:35,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:35,698 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:36,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:37,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:38,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:39,222 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:39,426 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:40,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:40,699 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:41,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:42,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:43,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:43,237 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:37:43,623 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:37:43,623 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:37:44,223 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:45,235 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:45,990 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:46,237 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:46,806 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:37:46,806 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 with policy now |
| 2025-01-15 21:37:46,809 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:37:46,810 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:37:46,810 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:37:46,810 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:37:47,028 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/91eblnjt-media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 |
| 2025-01-15 21:37:47,225 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:47,430 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:47,430 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:37:47,430 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 |
| 2025-01-15 21:37:47,430 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:37:47,692 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/xz27t7gr-media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 |
| 2025-01-15 21:37:48,261 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:48,430 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:49,254 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:49,431 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:50,282 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:51,261 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:51,637 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:52,272 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:53,259 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:53,432 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:54,256 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:55,287 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:55,433 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:37:56,292 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:57,009 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:37:57,306 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:58,313 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:58,630 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:37:58,630 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:37:58,631 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:37:58,661 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:37:58,661 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:37:59,329 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:37:59,435 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:37:59,435 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:38:00,312 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:01,334 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:01,436 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:02,359 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:02,877 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:03,313 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:04,332 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:05,346 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:05,437 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:06,378 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:07,350 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:08,376 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:08,719 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:09,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:10,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:11,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:11,440 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:12,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:13,238 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:38:13,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:13,626 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:38:13,627 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:38:13,784 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:14,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:15,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:16,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:17,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:18,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:19,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:19,717 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:20,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:21,348 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:22,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:23,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:24,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:24,718 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:25,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:26,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:27,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:28,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:28,627 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:38:28,627 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:38:29,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:29,762 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:30,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:31,349 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:32,350 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:33,350 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:34,350 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:35,350 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:35,719 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:36,357 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:37,358 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:37,451 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:37,505 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:38:37,505 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:38:37,506 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 with policy now |
| 2025-01-15 21:38:37,506 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:38:37,506 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:38:37,506 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:38:37,770 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ezlyee49-media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 |
| 2025-01-15 21:38:38,392 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:38,451 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:38:38,451 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 |
| 2025-01-15 21:38:38,452 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:38:38,733 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/83pkxwjo-media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 |
| 2025-01-15 21:38:39,409 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:39,452 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:40,380 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:40,452 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:40,737 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:41,380 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:42,373 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:43,238 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:38:43,379 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:43,453 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:43,659 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:38:43,659 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:38:44,383 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:45,408 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:46,421 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:46,433 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:47,417 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:47,455 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:48,440 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:49,446 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:50,421 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:51,437 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:51,753 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:52,446 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:53,434 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:54,440 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:54,458 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:38:54,701 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:38:54,701 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:38:54,701 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:38:55,436 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:55,458 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:38:55,459 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:38:56,480 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:57,447 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:57,459 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:38:57,573 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:38:58,470 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:38:58,657 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:38:58,658 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:38:59,454 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:00,465 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:01,449 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:01,461 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:02,493 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:02,750 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:03,464 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:04,460 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:05,460 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:05,463 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:06,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:07,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:07,465 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:08,136 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:08,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:09,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:10,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:11,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:12,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:13,239 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:39:13,240 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:13,461 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:13,630 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:39:13,631 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:39:14,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:15,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:16,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:17,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:17,470 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:18,325 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:18,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:19,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:20,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:21,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:22,462 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:23,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:23,748 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:24,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:25,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:26,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:27,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:28,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:28,631 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:39:28,631 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:39:28,793 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:29,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:30,463 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:31,478 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:32,351 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:39:32,351 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 with policy now |
| 2025-01-15 21:39:32,357 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:39:32,358 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:39:32,358 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:39:32,367 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:39:32,476 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:39:32,476 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 |
| 2025-01-15 21:39:32,476 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:39:32,477 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:32,576 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/q6owqepi-media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 |
| 2025-01-15 21:39:32,836 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/djti9mvf-media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 |
| 2025-01-15 21:39:33,477 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:33,510 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:34,477 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:34,523 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:34,782 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:35,477 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:35,493 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:36,497 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:37,478 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:37,521 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:38,496 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:39,491 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:40,353 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:40,509 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:41,480 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:41,520 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:42,531 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:43,239 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:39:43,527 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:43,651 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:39:43,651 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:39:44,530 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:45,534 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:45,619 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:45,759 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:39:45,759 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:39:45,759 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:39:46,482 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:39:46,482 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:39:46,524 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:46,759 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ks6w7yv7-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:39:47,483 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:47,530 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:48,558 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:49,483 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:49,577 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:50,550 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:51,116 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:51,484 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:39:51,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:52,561 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:53,564 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:54,594 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:55,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:56,597 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:56,806 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:39:57,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:58,614 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:39:58,652 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:39:58,653 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:39:59,590 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:00,581 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:01,489 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:01,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:01,960 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:02,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:03,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:04,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:05,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:06,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:07,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:07,781 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:08,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:09,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:10,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:11,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:12,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:12,781 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:13,239 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:40:13,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:13,638 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:40:13,638 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:40:14,583 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:15,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:16,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:17,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:17,782 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:18,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:19,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:20,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:21,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:22,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:22,782 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:23,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:24,584 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:25,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:26,585 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:27,587 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:28,020 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:28,365 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:40:28,365 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 with policy now |
| 2025-01-15 21:40:28,379 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:40:28,380 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:40:28,380 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:40:28,380 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:40:28,501 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:40:28,501 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 |
| 2025-01-15 21:40:28,501 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:40:28,593 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/poyozqqh-media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 |
| 2025-01-15 21:40:28,619 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:28,687 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:40:28,688 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:40:28,858 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/o8xocbsf-media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 |
| 2025-01-15 21:40:29,501 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:29,613 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:30,501 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:30,624 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:31,502 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:31,665 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:32,646 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:33,141 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:33,503 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:33,641 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:34,641 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:35,632 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:36,638 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:37,505 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:37,673 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:38,385 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:38,666 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:39,680 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:40,646 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:41,677 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:42,664 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:43,240 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:40:43,415 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:43,507 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:40:43,670 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:40:43,671 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:40:43,683 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:43,832 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:40:43,832 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:40:43,833 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:40:44,508 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:40:44,508 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:40:44,690 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:45,508 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:45,705 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:46,711 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:47,509 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:47,726 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:48,692 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:48,807 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:49,510 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:49,700 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:50,710 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:51,511 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:51,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:52,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:53,736 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:53,807 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:40:54,709 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:55,737 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:56,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:57,513 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:57,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:58,643 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:40:58,643 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:40:58,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:59,514 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:40:59,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:40:59,801 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:00,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:01,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:02,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:03,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:04,712 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:05,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:05,802 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:06,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:07,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:08,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:09,519 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:09,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:10,713 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:10,802 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:11,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:12,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:13,240 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:41:13,521 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:13,643 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:41:13,643 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:41:13,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:14,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:15,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:16,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:16,802 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:17,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:18,714 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:19,715 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:20,716 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:21,716 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:21,803 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:22,716 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:23,734 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:24,746 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:24,948 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:41:24,948 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 with policy now |
| 2025-01-15 21:41:24,952 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:41:24,952 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:41:24,952 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:41:24,953 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:41:25,181 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/fyspcu3h-media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 |
| 2025-01-15 21:41:25,526 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:25,526 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:41:25,526 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 |
| 2025-01-15 21:41:25,526 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:41:25,747 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:25,797 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ev5j11ae-media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 |
| 2025-01-15 21:41:26,526 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:26,754 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:27,063 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:27,527 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:27,790 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:28,703 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:41:28,703 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:41:28,811 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:29,783 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:30,784 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:31,528 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:31,820 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:32,070 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:32,800 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:33,529 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:33,779 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:34,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:35,530 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:41:35,804 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:35,898 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:41:35,898 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:41:35,898 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:41:36,531 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:41:36,531 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:41:36,811 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:37,531 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:37,745 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:37,817 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:38,816 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:39,532 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:39,802 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:40,814 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:41,533 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:41,809 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:42,842 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:43,089 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:43,241 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:41:43,693 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:41:43,693 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:41:43,859 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:44,835 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:45,855 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:46,855 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:47,873 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:48,634 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:48,836 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:49,831 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:50,832 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:51,538 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:41:51,831 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:52,831 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:53,811 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:53,832 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:54,832 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:55,832 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:56,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:57,832 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:58,648 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:41:58,649 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:41:58,813 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:41:58,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:41:59,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:00,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:01,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:02,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:03,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:04,811 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:04,833 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:05,857 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:06,846 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:07,277 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:42:07,277 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 with policy now |
| 2025-01-15 21:42:07,296 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:42:07,296 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:42:07,296 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:07,297 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:07,489 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/5qnttwb1-media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 |
| 2025-01-15 21:42:07,545 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:07,545 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:07,545 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 |
| 2025-01-15 21:42:07,546 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval |
| 2025-01-15 21:42:07,806 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/t7o4qhg3-media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 |
| 2025-01-15 21:42:07,856 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:08,546 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:08,886 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:09,546 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:09,850 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:09,858 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:10,875 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:11,859 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:12,873 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:13,241 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:42:13,548 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:13,676 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:42:13,676 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:42:13,890 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:14,892 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:15,637 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:15,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:16,891 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:17,550 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:17,909 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:17,947 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:42:17,948 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:17,948 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:18,550 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:42:18,550 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:18,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:19,174 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-15 21:42:19,174 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-15 21:42:19,327 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:42:19,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:19,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json |
| 2025-01-15 21:42:19,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media |
| 2025-01-15 21:42:19,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table |
| 2025-01-15 21:42:19,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval |
| 2025-01-15 21:42:19,638 INFO wandb-upload_3:2467459 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppigqiw9w |
| 2025-01-15 21:42:19,709 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-15 21:42:19,720 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-15 21:42:19,767 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/fz6rnj1y-media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json |
| 2025-01-15 21:42:19,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:20,085 INFO SenderThread:2467459 [sender.py:send_request_log_artifact():1525] logged artifact run-vfeb54wi-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0NDA3MTY2OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzOTEzNDQyMA==', 'latestArtifact': None}} |
| 2025-01-15 21:42:20,085 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:42:20,085 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json with policy now |
| 2025-01-15 21:42:20,085 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:42:20,085 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,086 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,086 DEBUG SenderThread:2467459 [sender.py:send():379] send: telemetry |
| 2025-01-15 21:42:20,086 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,090 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,090 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,091 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,091 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,091 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,091 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,092 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,092 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,092 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,092 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,093 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,093 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,093 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,094 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,094 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,094 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,095 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,095 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,095 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,095 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:20,096 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:20,096 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-15 21:42:20,334 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ah1sq9kq-media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json |
| 2025-01-15 21:42:20,551 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:42:20,552 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:20,552 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json |
| 2025-01-15 21:42:20,552 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval |
| 2025-01-15 21:42:20,595 INFO wandb-upload_3:2467459 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpknxrse51 |
| 2025-01-15 21:42:20,741 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/vxvgjgrh-media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json |
| 2025-01-15 21:42:20,799 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/ttd8s49d-events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:42:20,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:21,036 INFO SenderThread:2467459 [sender.py:send_request_log_artifact():1525] logged artifact run-vfeb54wi-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0NDA3MTY3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzOTEzNDQyMQ==', 'latestArtifact': None}} |
| 2025-01-15 21:42:21,037 DEBUG SenderThread:2467459 [sender.py:send():379] send: files |
| 2025-01-15 21:42:21,037 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:21,037 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json with policy now |
| 2025-01-15 21:42:21,037 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:42:21,037 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:21,038 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:21,280 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /tmp/tmp_oyimap2wandb/v1krkjej-media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json |
| 2025-01-15 21:42:21,552 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:21,552 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:21,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:22,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:23,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:24,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:25,902 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:26,820 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:26,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:27,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:28,649 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:42:28,649 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:42:28,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:29,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:30,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:31,823 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:31,918 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:32,557 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/config.yaml |
| 2025-01-15 21:42:32,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:33,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:34,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:35,903 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:36,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:37,820 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:37,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:38,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:39,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:40,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:41,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:42,821 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:42,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:43,241 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:42:43,649 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-15 21:42:43,649 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-15 21:42:43,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:44,904 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:45,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:46,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:47,821 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:47,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:48,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:49,905 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:42:50,846 DEBUG SenderThread:2467459 [sender.py:send():379] send: exit |
| 2025-01-15 21:42:50,846 INFO SenderThread:2467459 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-15 21:42:50,846 INFO SenderThread:2467459 [sender.py:send_exit():588] handling runtime: 1057 |
| 2025-01-15 21:42:50,846 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:50,846 INFO SenderThread:2467459 [sender.py:send_exit():594] send defer |
| 2025-01-15 21:42:50,846 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:50,846 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-15 21:42:50,847 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:50,847 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-15 21:42:50,847 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-15 21:42:50,847 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:50,847 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-15 21:42:50,847 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:50,847 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-15 21:42:50,847 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-15 21:42:50,847 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:50,847 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-15 21:42:50,847 INFO HandlerThread:2467459 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-15 21:42:50,847 DEBUG SystemMonitor:2467459 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-15 21:42:50,847 DEBUG SystemMonitor:2467459 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-15 21:42:50,849 INFO HandlerThread:2467459 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-15 21:42:50,849 INFO HandlerThread:2467459 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-15 21:42:50,913 INFO HandlerThread:2467459 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-15 21:42:50,913 INFO HandlerThread:2467459 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-15 21:42:50,913 INFO HandlerThread:2467459 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-15 21:42:50,914 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:50,914 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-15 21:42:50,914 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-15 21:42:50,914 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:50,914 DEBUG SenderThread:2467459 [sender.py:send():379] send: stats |
| 2025-01-15 21:42:50,914 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-15 21:42:50,915 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:50,915 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-15 21:42:50,915 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-15 21:42:50,916 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:50,916 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-15 21:42:51,567 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:53,568 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:42:56,958 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:42:56,958 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:56,958 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-15 21:42:56,959 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:56,959 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-15 21:42:56,959 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-15 21:42:56,960 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:42:56,960 DEBUG SenderThread:2467459 [sender.py:send():379] send: history |
| 2025-01-15 21:42:56,960 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-15 21:42:56,960 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:56,960 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-15 21:42:56,961 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:56,961 DEBUG SenderThread:2467459 [sender.py:send():379] send: summary |
| 2025-01-15 21:42:56,961 INFO SenderThread:2467459 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-15 21:42:56,962 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:56,962 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-15 21:42:56,962 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-15 21:42:56,962 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:56,962 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-15 21:42:56,962 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:56,962 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-15 21:42:56,962 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-15 21:42:56,962 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-15 21:42:56,962 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:56,962 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-15 21:42:56,962 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:56,962 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-15 21:42:57,570 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:42:57,847 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:42:59,953 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-15 21:42:59,953 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:42:59,954 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:59,954 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-15 21:42:59,954 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:42:59,954 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-15 21:42:59,954 INFO SenderThread:2467459 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-15 21:42:59,957 INFO SenderThread:2467459 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-15 21:42:59,959 INFO SenderThread:2467459 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-15 21:42:59,976 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-15 21:42:59,976 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:42:59,976 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-15 21:42:59,976 DEBUG SenderThread:2467459 [sender.py:send():379] send: artifact |
| 2025-01-15 21:43:00,572 INFO Thread-12 :2467459 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:43:00,626 INFO wandb-upload_0:2467459 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmphw5sjkl5 |
| 2025-01-15 21:43:00,848 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:43:00,891 INFO wandb-upload_2:2467459 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpk_xp6b81 |
| 2025-01-15 21:43:00,941 INFO wandb-upload_3:2467459 [upload_job.py:push():88] Uploaded file /tmp/tmpx6bsw9uf/wandb-job.json |
| 2025-01-15 21:43:01,577 INFO SenderThread:2467459 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ0NDA3MjMyOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjU1MzUzMw==', 'versionIndex': 439}}} |
| 2025-01-15 21:43:01,577 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:01,577 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-15 21:43:01,577 INFO SenderThread:2467459 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/config.yaml config.yaml |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json wandb-summary.json |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 events.out.tfevents.1736994313.leonmkim-ROG-Strix-G15CS-G15CS.2467097.0 |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/requirements.txt requirements.txt |
| 2025-01-15 21:43:02,573 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-15 21:43:02,574 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-15 21:43:02,574 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log output.log |
| 2025-01-15 21:43:02,574 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/diff.patch diff.patch |
| 2025-01-15 21:43:02,575 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-15 21:43:02,576 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 media/videos/eval/15_eval_15_57fbf8f750a05cbd03dd.mp4 |
| 2025-01-15 21:43:02,576 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 media/videos/eval/11_eval_11_1f892c1875d8f769e839.mp4 |
| 2025-01-15 21:43:02,576 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 media/videos/eval/5_eval_5_7562c03fef4135eb5f56.mp4 |
| 2025-01-15 21:43:02,577 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 media/videos/eval/19_eval_19_c347402d83b94d0d76b5.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 media/videos/eval/17_eval_17_8e0251ea6b3f5a1ef81f.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 media/videos/eval/14_eval_14_389de3f12b22a271b1c9.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 media/videos/eval/6_eval_6_54affceed9218be400c4.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 media/videos/eval/3_eval_3_327a1cef6154eb393524.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 media/videos/eval/2_eval_2_bb336114555212728d0b.mp4 |
| 2025-01-15 21:43:02,580 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 media/videos/eval/12_eval_12_b7f16c34c55c022ec9cf.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 media/videos/eval/16_eval_16_c066e7ff5f521f1bd1c3.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 media/videos/eval/13_eval_13_b286629d17bfd49bd46b.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 media/videos/eval/18_eval_18_792d83e6a625ca80ced8.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 media/videos/eval/1_eval_1_4b58a2f9660a5285887a.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 media/videos/eval/9_eval_9_9f85b9e56cea35531fc0.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 media/videos/eval/10_eval_10_be070cda4484f5d8f382.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 media/videos/eval/4_eval_4_65dbd3f5364b11e95697.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 media/videos/eval/0_eval_0_3d1b728d66af3d385c76.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 media/videos/eval/8_eval_8_2ccbfe16dcecf0f3420f.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 media/videos/eval/7_eval_7_8602e79d56991114b037.mp4 |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json media/table/eval/total_success_rate_ci_21_c6c1470740707d609aca.table.json |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json media/table/eval/success_rate_ci_20_c4da0e063c4adfc6cc66.table.json |
| 2025-01-15 21:43:02,581 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-15 21:43:02,582 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:43:02,582 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:43:02,582 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-15 21:43:02,582 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:02,582 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-15 21:43:02,582 INFO SenderThread:2467459 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-15 21:43:02,756 INFO wandb-upload_0:2467459 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/wandb-summary.json |
| 2025-01-15 21:43:02,759 INFO wandb-upload_2:2467459 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/requirements.txt |
| 2025-01-15 21:43:02,760 INFO wandb-upload_1:2467459 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/config.yaml |
| 2025-01-15 21:43:02,789 INFO wandb-upload_3:2467459 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/files/output.log |
| 2025-01-15 21:43:02,849 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:43:02,849 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:43:02,989 INFO Thread-11 (_thread_body):2467459 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-15 21:43:02,989 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:43:02,989 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-15 21:43:02,989 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:02,990 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-15 21:43:02,990 INFO SenderThread:2467459 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-15 21:43:02,993 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-15 21:43:02,993 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:43:02,993 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-15 21:43:02,993 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:02,993 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-15 21:43:02,993 INFO SenderThread:2467459 [file_stream.py:finish():601] file stream finish called |
| 2025-01-15 21:43:03,229 INFO SenderThread:2467459 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-15 21:43:03,229 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-15 21:43:03,229 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:43:03,229 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-15 21:43:03,229 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:03,229 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-15 21:43:03,229 INFO SenderThread:2467459 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-15 21:43:03,229 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-15 21:43:03,230 INFO HandlerThread:2467459 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-15 21:43:03,230 DEBUG SenderThread:2467459 [sender.py:send():379] send: final |
| 2025-01-15 21:43:03,230 DEBUG SenderThread:2467459 [sender.py:send():379] send: footer |
| 2025-01-15 21:43:03,230 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: defer |
| 2025-01-15 21:43:03,230 INFO SenderThread:2467459 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-15 21:43:03,230 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:43:03,231 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-15 21:43:03,231 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:43:03,231 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: server_info |
| 2025-01-15 21:43:03,235 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-15 21:43:03,235 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-15 21:43:03,236 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-15 21:43:03,237 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-15 21:43:03,295 DEBUG SenderThread:2467459 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-15 21:43:03,296 INFO MainThread:2467459 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-15 21:43:03,296 INFO MainThread:2467459 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-15 21:43:03,297 INFO MainThread:2467459 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-15 21:43:03,297 DEBUG HandlerThread:2467459 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-15 21:43:03,297 INFO HandlerThread:2467459 [handler.py:finish():882] shutting down handler |
| 2025-01-15 21:43:04,235 INFO WriterThread:2467459 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/vfeb54wi/wandb/run-20250115_212512-vfeb54wi/run-vfeb54wi.wandb |
| 2025-01-15 21:43:04,296 INFO SenderThread:2467459 [sender.py:finish():1615] shutting down sender |
| 2025-01-15 21:43:04,296 INFO SenderThread:2467459 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-15 21:43:04,296 INFO SenderThread:2467459 [file_pusher.py:join():175] waiting for file pusher |
|
|
