| 2025-01-17 14:47:41,921 INFO StreamThr :2517552 [internal.py:wandb_internal():85] W&B internal server running at pid: 2517552, started at: 2025-01-17 14:47:41.921158 |
| 2025-01-17 14:47:41,922 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status |
| 2025-01-17 14:47:41,933 INFO WriterThread:2517552 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/run-up07mbix.wandb |
| 2025-01-17 14:47:41,934 DEBUG SenderThread:2517552 [sender.py:send():379] send: header |
| 2025-01-17 14:47:41,942 DEBUG SenderThread:2517552 [sender.py:send():379] send: run |
| 2025-01-17 14:47:42,165 INFO SenderThread:2517552 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files |
| 2025-01-17 14:47:42,165 INFO SenderThread:2517552 [sender.py:_start_run_threads():1188] run started: up07mbix with start time 1737143261.921632 |
| 2025-01-17 14:47:42,181 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-17 14:47:42,182 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: check_version |
| 2025-01-17 14:47:42,265 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-17 14:47:42,270 DEBUG HandlerThread:2517552 [system_info.py:__init__():26] System info init |
| 2025-01-17 14:47:42,270 DEBUG HandlerThread:2517552 [system_info.py:__init__():41] System info init done |
| 2025-01-17 14:47:42,270 INFO HandlerThread:2517552 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-17 14:47:42,270 INFO SystemMonitor:2517552 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-17 14:47:42,270 INFO HandlerThread:2517552 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-17 14:47:42,271 INFO SystemMonitor:2517552 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-17 14:47:42,271 INFO SystemMonitor:2517552 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-17 14:47:42,272 INFO SystemMonitor:2517552 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-17 14:47:42,272 INFO SystemMonitor:2517552 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-17 14:47:42,273 INFO SystemMonitor:2517552 [interfaces.py:start():188] Started network monitoring |
| 2025-01-17 14:47:42,282 DEBUG HandlerThread:2517552 [system_info.py:probe():152] Probing system |
| 2025-01-17 14:47:42,286 DEBUG HandlerThread:2517552 [system_info.py:_probe_git():137] Probing git |
| 2025-01-17 14:47:42,292 DEBUG HandlerThread:2517552 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-17 14:47:42,293 DEBUG HandlerThread:2517552 [system_info.py:probe():200] Probing system done |
| 2025-01-17 14:47:42,293 DEBUG HandlerThread:2517552 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-17T19:47:42.282982', 'startedAt': '2025-01-17T19:47:41.909118', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4134.258, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.962, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 399.98992919921875}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-17 14:47:42,293 INFO HandlerThread:2517552 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-17 14:47:42,293 INFO HandlerThread:2517552 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-17 14:47:42,293 DEBUG HandlerThread:2517552 [system_info.py:_save_code():44] Saving code |
| 2025-01-17 14:47:42,294 DEBUG HandlerThread:2517552 [system_info.py:_save_code():67] Saving code done |
| 2025-01-17 14:47:42,294 DEBUG HandlerThread:2517552 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-17 14:47:42,390 DEBUG HandlerThread:2517552 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-17 14:47:42,390 INFO HandlerThread:2517552 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-17 14:47:42,393 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:47:42,393 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-17 14:47:42,394 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-17 14:47:42,394 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-17 14:47:42,394 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-17 14:47:42,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-17 14:47:42,626 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-17 14:47:42,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:47:42,627 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:47:42,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:42,655 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/5fdjf9gk-code/FISH/eval_robot.py |
| 2025-01-17 14:47:42,664 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/9ted8sye-wandb-metadata.json |
| 2025-01-17 14:47:42,669 INFO HandlerThread:2517552 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/tb" |
| save: true |
| } |
| control { |
| relay_id: "ac8462253926464da2d1680d5c59cfdf" |
| } |
| _info { |
| stream_id: "up07mbix" |
| } |
|
|
| 2025-01-17 14:47:42,669 DEBUG HandlerThread:2517552 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-17 14:47:42,688 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/aoalkeoe-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 14:47:42,689 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/kci0a5wf-diff.patch |
| 2025-01-17 14:47:42,691 DEBUG SenderThread:2517552 [sender.py:send():379] send: telemetry |
| 2025-01-17 14:47:42,737 DEBUG SenderThread:2517552 [sender.py:send():379] send: tbrecord |
| 2025-01-17 14:47:42,738 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:47:42,738 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 with policy live |
| 2025-01-17 14:47:42,919 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/85ejh0y0-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:47:43,166 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-metadata.json |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/requirements.txt |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/diff.patch |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/code/FISH/eval_robot.py |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/code/FISH |
| 2025-01-17 14:47:43,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/code |
| 2025-01-17 14:47:43,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:44,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:45,166 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:45,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:46,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:47,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:47:47,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:48,084 DEBUG SenderThread:2517552 [sender.py:send():379] send: config |
| 2025-01-17 14:47:48,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:49,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:49,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:50,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:51,167 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:51,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:52,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:47:52,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:53,168 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:53,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:54,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:55,169 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:55,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:56,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:57,169 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:47:57,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:47:57,626 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:47:57,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:57,715 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:47:58,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:47:59,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:00,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:01,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:02,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:03,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:03,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:04,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:05,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:06,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:07,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:08,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:08,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:09,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:10,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:11,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:12,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:48:12,626 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:48:12,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:13,174 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:13,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:13,736 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:14,174 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/config.yaml |
| 2025-01-17 14:48:14,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:15,174 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:15,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:16,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:17,175 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:17,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:18,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:19,628 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:19,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:20,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:21,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:22,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:23,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:24,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:25,473 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:25,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:26,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:27,178 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:27,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:48:27,626 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:48:27,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:28,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:29,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:30,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:30,776 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:31,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:32,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:33,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:34,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:35,180 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:35,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:36,317 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:36,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:37,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:38,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:39,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:40,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:41,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:41,983 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:42,172 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:48:42,172 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 with policy now |
| 2025-01-17 14:48:42,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:48:42,174 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:48:42,174 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:48:42,174 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:48:42,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:48:42,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 |
| 2025-01-17 14:48:42,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos |
| 2025-01-17 14:48:42,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:48:42,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media |
| 2025-01-17 14:48:42,273 DEBUG SystemMonitor:2517552 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-17 14:48:42,274 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:48:42,469 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ybp3svcy-media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 |
| 2025-01-17 14:48:42,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:48:42,627 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:48:42,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:42,745 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/lbxqg749-media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 |
| 2025-01-17 14:48:43,182 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:43,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:44,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:45,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:46,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:47,184 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:47,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:47,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:48,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:49,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:50,636 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:51,185 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:51,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:52,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:53,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:53,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:54,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:55,186 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:48:55,444 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/a46a0ts1-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:48:55,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:56,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:57,187 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:57,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:48:57,627 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:48:57,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:58,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:48:59,188 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:48:59,383 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:48:59,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:00,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:01,188 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:01,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:02,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:03,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:04,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:04,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:05,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:06,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:07,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:08,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:09,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:09,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:10,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:11,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:12,283 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:49:12,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:49:12,627 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:49:12,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:13,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:14,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:14,751 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:15,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:16,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:17,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:18,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:19,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:20,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:20,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:21,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:22,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:23,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:24,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:25,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:25,862 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:26,270 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:49:26,270 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 with policy now |
| 2025-01-17 14:49:26,273 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:49:26,273 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:49:26,274 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:49:26,274 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:49:26,525 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/hg6wq3ey-media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 |
| 2025-01-17 14:49:26,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:27,198 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:49:27,198 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:27,198 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 |
| 2025-01-17 14:49:27,198 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:49:27,484 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/b3bqb9i9-media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 |
| 2025-01-17 14:49:27,634 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:49:27,634 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:49:27,677 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:28,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:29,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:30,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:30,937 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:31,199 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:31,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:32,665 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:33,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:34,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:35,201 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:35,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:36,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:36,666 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:37,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:38,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:39,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:40,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:41,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:42,259 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:42,275 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:49:42,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:49:42,629 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:49:42,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:43,142 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:49:43,142 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:49:43,142 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:49:43,203 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:49:43,204 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:43,204 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:49:43,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:44,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:45,204 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:45,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:46,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:47,205 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:49:47,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:47,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:48,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:49,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:50,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:51,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:52,632 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:52,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:53,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:54,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:55,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:56,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:57,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:49:57,629 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:49:57,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:57,740 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:49:58,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:49:59,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:00,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:01,210 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:01,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:02,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:03,211 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:03,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:03,656 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:04,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:05,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:06,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:07,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:08,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:09,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:09,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:10,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:11,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:12,275 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:50:12,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:50:12,629 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:50:12,677 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:13,662 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:14,407 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:50:14,408 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 with policy now |
| 2025-01-17 14:50:14,410 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:50:14,410 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:50:14,411 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:50:14,411 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:50:14,658 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:14,665 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/cnvywjy0-media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 |
| 2025-01-17 14:50:15,216 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:50:15,216 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:15,216 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 |
| 2025-01-17 14:50:15,217 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:50:15,428 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:15,501 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/kgs37oz6-media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 |
| 2025-01-17 14:50:15,660 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:16,661 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:17,217 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:17,669 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:18,662 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:19,217 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:19,671 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:20,675 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:20,823 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:21,676 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:22,675 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:23,219 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:23,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:24,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:25,676 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:26,643 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:26,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:27,220 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:50:27,490 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/mkcw0g0c-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:50:27,633 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:50:27,633 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:50:27,677 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:27,938 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:50:27,938 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:50:27,938 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:50:28,221 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:50:28,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:29,221 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:29,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:30,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:31,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:32,139 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:32,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:33,222 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:33,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:34,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:35,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:36,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:37,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:37,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:38,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:39,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:40,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:41,225 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:41,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:42,276 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:50:42,629 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:50:42,630 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:50:42,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:42,726 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:43,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:44,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:45,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:46,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:47,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:48,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:48,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:49,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:50,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:51,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:52,682 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:53,682 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:54,638 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:50:54,682 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:55,682 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:56,683 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:57,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:50:57,630 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:50:57,683 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:58,689 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:50:59,137 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:50:59,137 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:50:59,138 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 with policy now |
| 2025-01-17 14:50:59,138 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:50:59,138 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:50:59,138 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:50:59,232 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:50:59,232 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:50:59,232 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 |
| 2025-01-17 14:50:59,233 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:50:59,390 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ma9sq65m-media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 |
| 2025-01-17 14:50:59,665 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/8lbemu2o-media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 |
| 2025-01-17 14:50:59,691 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:00,165 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:00,693 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:01,233 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:01,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:02,690 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:03,685 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:04,691 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:05,234 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:05,565 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:05,687 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:06,688 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:07,235 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:07,687 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:08,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:09,695 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:10,642 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:10,695 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:11,695 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:11,992 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:51:11,993 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:51:11,993 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:51:12,237 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:51:12,237 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:51:12,276 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:51:12,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:51:12,641 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:51:12,692 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:13,237 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:13,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:14,699 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:15,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:16,546 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:16,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:17,239 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:17,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:18,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:19,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:20,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:21,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:21,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:22,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:23,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:24,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:25,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:26,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:26,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:27,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:51:27,630 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:51:27,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:28,696 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:29,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:30,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:31,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:31,740 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:32,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:33,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:34,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:35,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:36,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:37,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:37,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:38,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:39,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:40,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:41,698 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:42,276 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:51:42,631 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:51:42,631 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:51:42,706 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:42,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:43,248 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:43,381 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:51:43,382 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 with policy now |
| 2025-01-17 14:51:43,386 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:51:43,387 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:51:43,387 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:51:43,388 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:51:43,632 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ixvd3p6b-media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 |
| 2025-01-17 14:51:43,708 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:44,248 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:51:44,248 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 |
| 2025-01-17 14:51:44,248 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:51:44,551 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/qmgc5aka-media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 |
| 2025-01-17 14:51:44,708 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:45,248 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:45,715 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:46,713 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:47,712 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:48,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:48,717 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:49,250 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:49,718 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:50,715 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:51,251 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:51,718 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:52,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:53,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:54,164 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:51:54,252 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:51:54,512 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/r2k4us42-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:51:54,725 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:55,044 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:51:55,045 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:51:55,045 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:51:55,252 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:51:55,252 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:55,718 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:56,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:57,253 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:57,637 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:51:57,638 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:51:57,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:58,733 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:59,253 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:51:59,727 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:51:59,964 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:00,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:01,254 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:01,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:02,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:03,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:04,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:05,643 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:05,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:06,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:07,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:08,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:09,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:10,644 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:10,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:11,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:12,277 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:52:12,630 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:52:12,631 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:52:12,721 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:13,721 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:14,721 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:15,721 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:15,789 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:16,721 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:17,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:18,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:19,723 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:20,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:21,644 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:21,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:22,722 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:23,723 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:24,723 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:25,723 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:26,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:26,723 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:27,643 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:52:27,643 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:52:27,738 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:28,247 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:52:28,247 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 with policy now |
| 2025-01-17 14:52:28,254 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:52:28,255 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:52:28,255 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:52:28,255 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:52:28,264 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:52:28,264 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 |
| 2025-01-17 14:52:28,265 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:52:28,504 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/qdc9zoja-media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 |
| 2025-01-17 14:52:28,740 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:28,766 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/o5g4fsr7-media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 |
| 2025-01-17 14:52:29,265 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:29,738 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:30,737 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:31,736 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:31,925 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:32,749 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:33,266 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:33,744 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:34,751 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:35,750 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:36,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:37,268 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:37,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:37,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:38,753 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:39,763 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:40,107 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:52:40,107 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:52:40,108 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:52:40,269 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:52:40,269 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:52:40,763 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:41,269 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:41,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:42,277 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:52:42,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:52:42,640 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:52:42,765 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:42,777 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:43,270 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:43,758 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:44,759 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:45,270 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:45,759 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:46,759 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:47,759 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:48,088 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:48,759 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:49,272 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:49,760 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:50,760 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:51,273 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:51,760 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:52,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:53,274 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:52:53,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:53,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:54,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:55,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:56,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:57,635 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:52:57,635 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:52:57,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:58,749 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:52:58,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:52:59,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:00,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:01,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:02,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:03,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:04,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:04,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:05,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:06,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:07,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:08,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:09,763 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:10,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:10,763 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:11,769 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:12,129 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:53:12,129 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 with policy now |
| 2025-01-17 14:53:12,137 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:53:12,138 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:53:12,138 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:53:12,141 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:53:12,278 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:53:12,281 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:53:12,282 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 |
| 2025-01-17 14:53:12,282 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:53:12,382 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/md47hjpp-media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 |
| 2025-01-17 14:53:12,644 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:53:12,645 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:53:12,665 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/6upv9n67-media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 |
| 2025-01-17 14:53:12,780 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:13,282 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:13,773 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:14,784 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:15,283 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:15,784 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:15,786 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:16,791 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:17,283 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:17,801 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:18,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:19,796 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:20,784 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:20,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:21,285 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:21,790 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:22,285 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:53:22,538 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/b3hid05a-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:53:22,801 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:23,161 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:53:23,161 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:53:23,161 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:53:23,286 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:53:23,286 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:23,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:24,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:25,286 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:25,801 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:26,195 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:26,796 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:27,287 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:27,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:53:27,639 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:53:27,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:28,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:29,288 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:29,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:30,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:31,713 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:31,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:32,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:33,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:34,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:35,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:36,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:37,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:37,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:38,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:39,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:40,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:41,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:42,278 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:53:42,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:53:42,640 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:53:42,796 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:42,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:43,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:44,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:45,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:46,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:47,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:48,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:48,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:49,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:50,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:51,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:52,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:53,822 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:54,197 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:54,587 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:53:54,587 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 with policy now |
| 2025-01-17 14:53:54,590 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:53:54,590 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:53:54,590 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:53:54,590 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:53:54,810 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:54,847 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ls4gvw2l-media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 |
| 2025-01-17 14:53:55,298 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:53:55,298 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:55,298 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 |
| 2025-01-17 14:53:55,298 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:53:55,578 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/pjw07pfa-media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 |
| 2025-01-17 14:53:55,816 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:56,827 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:57,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:53:57,652 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:53:57,840 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:58,839 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:53:59,299 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:53:59,317 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:53:59,842 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:00,845 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:01,838 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:02,847 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:03,301 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:03,836 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:04,357 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:04,844 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:05,213 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:54:05,213 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:54:05,213 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:54:05,302 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:54:05,302 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:05,302 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:54:05,846 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:06,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:07,302 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:07,845 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:08,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:09,303 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:09,429 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:09,857 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:10,865 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:11,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:12,279 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:54:12,649 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:54:12,649 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:54:12,851 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:13,304 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:13,851 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:14,851 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:14,915 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:15,305 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:15,852 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:16,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:17,306 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:17,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:18,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:19,307 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:19,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:20,689 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:20,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:21,308 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:21,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:22,853 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:23,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:24,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:25,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:26,661 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:26,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:27,649 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:54:27,649 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:54:27,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:28,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:29,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:30,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:31,726 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:31,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:32,854 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:33,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:34,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:35,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:36,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:37,661 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:37,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:38,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:39,855 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:40,873 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:41,194 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:54:41,194 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 with policy now |
| 2025-01-17 14:54:41,202 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:54:41,202 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:54:41,202 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:54:41,203 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:54:41,315 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:54:41,315 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:41,316 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 |
| 2025-01-17 14:54:41,316 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:54:41,459 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/71638wfj-media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 |
| 2025-01-17 14:54:41,740 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/yv16vlac-media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 |
| 2025-01-17 14:54:41,898 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:42,279 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:54:42,667 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:42,667 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:54:42,667 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:54:42,905 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:43,316 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:43,897 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:44,937 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:45,917 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:46,896 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:47,317 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:47,680 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:47,901 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:48,913 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:49,318 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:49,898 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:50,273 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:54:50,274 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:54:50,274 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:54:50,319 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:54:50,319 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:54:50,589 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/bkkmsaqw-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:54:50,900 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:51,319 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:51,905 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:52,903 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:53,644 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:53,901 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:54,904 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:55,321 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:55,916 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:56,921 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:57,668 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:54:57,668 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:54:57,922 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:58,705 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:54:58,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:54:59,322 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:54:59,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:00,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:01,323 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:01,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:02,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:03,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:04,668 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:04,906 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:05,907 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:06,907 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:07,907 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:08,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:09,668 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:09,907 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:10,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:11,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:12,280 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:55:12,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:55:12,650 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:55:12,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:13,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:14,731 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:14,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:15,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:16,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:17,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:18,908 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:19,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:19,948 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:20,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:21,331 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:21,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:22,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:23,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:24,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:25,520 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:25,934 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:26,898 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:55:26,898 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 with policy now |
| 2025-01-17 14:55:26,912 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:55:26,913 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:55:26,913 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:55:26,913 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:55:26,945 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:27,179 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/zay1i2n7-media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 |
| 2025-01-17 14:55:27,334 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:55:27,334 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:27,334 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 |
| 2025-01-17 14:55:27,334 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:55:27,614 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/a8q9styt-media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 |
| 2025-01-17 14:55:27,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:55:27,654 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:55:27,917 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:28,936 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:29,334 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:29,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:30,599 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:30,949 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:31,335 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:31,971 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:32,934 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:33,932 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:34,950 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:35,337 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:35,705 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:35,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:36,950 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:37,965 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:38,341 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:55:38,341 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:55:38,342 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:55:38,342 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:55:38,968 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:39,343 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:55:39,343 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:39,952 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:40,953 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:41,025 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:41,344 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:41,976 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:42,280 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:55:42,661 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:55:42,662 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:55:42,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:43,344 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:43,954 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:44,962 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:45,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:46,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:47,005 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:47,346 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:55:47,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:48,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:49,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:50,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:51,956 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:52,672 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:52,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:53,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:54,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:55,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:56,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:57,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:55:57,652 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:55:57,810 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:55:57,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:58,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:55:59,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:00,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:01,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:02,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:03,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:03,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:04,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:05,958 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:06,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:07,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:08,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:08,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:09,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:10,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:11,959 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:12,281 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:56:12,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:56:12,652 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:56:13,004 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:13,356 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:13,966 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:13,988 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:14,061 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:56:14,061 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 with policy now |
| 2025-01-17 14:56:14,064 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:56:14,064 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:56:14,064 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:56:14,065 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:56:14,336 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ct8nouz2-media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 |
| 2025-01-17 14:56:14,357 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:56:14,357 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 |
| 2025-01-17 14:56:14,357 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:56:14,632 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ucmgw36v-media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 |
| 2025-01-17 14:56:14,970 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:15,357 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:15,976 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:16,977 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:17,974 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:18,997 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:19,359 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:19,773 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:19,984 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:20,969 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:21,361 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:21,983 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:22,988 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:23,361 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:24,021 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:25,012 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:25,182 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:25,362 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:56:25,396 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:56:25,397 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:56:25,397 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:56:25,644 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/g762glol-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:56:26,012 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:26,363 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:56:27,022 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:27,363 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:27,654 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:56:27,654 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:56:28,011 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:29,016 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:30,013 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:30,428 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:31,048 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:31,365 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:32,043 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:33,051 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:34,051 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:35,056 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:35,483 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:36,044 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:37,044 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:37,367 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:38,044 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:39,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:40,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:40,689 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:41,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:42,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:42,289 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:56:42,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:56:42,652 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:56:43,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:44,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:45,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:45,370 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:56:46,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:46,545 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:47,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:48,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:49,045 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:50,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:51,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:51,690 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:52,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:53,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:54,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:55,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:56,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:56,690 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:56:57,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:57,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:56:57,652 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:56:58,047 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:56:59,047 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:00,047 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:01,047 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:01,725 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:02,047 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:03,068 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:03,378 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:03,754 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:57:03,754 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 with policy now |
| 2025-01-17 14:57:03,765 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:57:03,765 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:57:03,765 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:57:03,766 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:57:04,013 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/yw7m3f6j-media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 |
| 2025-01-17 14:57:04,088 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:04,378 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:57:04,378 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 |
| 2025-01-17 14:57:04,379 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:57:04,674 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/hssrhkv3-media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 |
| 2025-01-17 14:57:05,056 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:05,379 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:06,060 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:06,847 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:07,094 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:08,064 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:09,076 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:09,380 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:10,074 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:11,076 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:11,381 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:12,088 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:12,281 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:57:12,282 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:12,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:57:12,678 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:57:13,098 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:13,382 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:57:13,451 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:57:13,451 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:57:13,453 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:57:14,095 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:14,382 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:57:14,382 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:57:15,108 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:15,383 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:16,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:17,120 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:17,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:18,125 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:19,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:19,384 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:20,136 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:21,140 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:22,128 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:22,705 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:23,132 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:24,113 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:25,113 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:25,386 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:26,113 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:27,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:27,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:57:27,653 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:57:27,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:28,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:29,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:30,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:31,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:32,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:33,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:33,703 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:34,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:35,114 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:36,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:37,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:37,391 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:38,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:39,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:39,703 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:40,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:41,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:42,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:42,282 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:57:42,653 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:57:42,654 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:57:43,115 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:44,116 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:44,758 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:45,116 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:46,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:47,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:48,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:49,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:49,396 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:50,117 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:50,313 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:51,150 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:51,397 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:51,487 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:57:51,487 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 with policy now |
| 2025-01-17 14:57:51,493 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:57:51,493 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:57:51,493 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:57:51,494 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:57:51,748 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/hqx05krv-media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 |
| 2025-01-17 14:57:52,133 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:52,398 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:57:52,398 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 |
| 2025-01-17 14:57:52,398 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:57:52,707 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/u4scvtze-media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 |
| 2025-01-17 14:57:53,151 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:53,398 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:54,152 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:55,193 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:56,150 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:56,193 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:57:57,159 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:57,400 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:57:57,683 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:57:57,684 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:57:58,146 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:59,154 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:57:59,401 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:00,145 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:01,151 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:01,325 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:02,149 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:02,402 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:58:02,514 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:58:02,514 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:58:02,514 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:58:03,164 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:03,402 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:58:03,402 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:04,181 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:05,148 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:05,403 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:06,161 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:07,152 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:07,205 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:07,404 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:08,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:09,176 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:10,169 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:11,169 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:12,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:12,282 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:58:12,283 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:12,672 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:58:12,672 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:58:13,179 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:14,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:15,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:15,407 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:16,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:17,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:17,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:18,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:19,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:20,171 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:21,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:22,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:22,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:23,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:24,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:25,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:26,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:27,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:27,665 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:58:27,666 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:58:27,813 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:28,172 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:29,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:30,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:31,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:32,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:33,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:33,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:34,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:35,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:36,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:37,174 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:38,173 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:38,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:39,174 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:40,174 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:41,181 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:41,418 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:42,179 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:42,271 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:58:42,271 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 with policy now |
| 2025-01-17 14:58:42,283 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:58:42,285 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:58:42,285 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:58:42,285 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:58:42,286 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:58:42,418 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:58:42,419 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 |
| 2025-01-17 14:58:42,419 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:58:42,540 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/x4gcj7og-media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 |
| 2025-01-17 14:58:42,672 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:58:42,672 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:58:42,812 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/alyp4d5r-media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 |
| 2025-01-17 14:58:43,204 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:43,419 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:43,783 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:44,212 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:45,203 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:45,420 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:46,242 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:47,218 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:47,420 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:48,220 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:49,218 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:49,240 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:50,259 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:50,422 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:58:50,576 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:58:50,576 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:58:50,581 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:58:50,695 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/jbot3eff-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:58:51,233 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:51,422 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:58:51,422 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:51,422 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:58:52,269 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:53,263 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:53,423 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:54,236 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:54,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:58:55,259 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:55,424 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:58:56,261 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:57,238 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:57,686 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:58:57,686 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:58:58,235 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:59,262 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:58:59,844 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:00,288 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:01,258 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:02,265 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:03,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:03,427 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:04,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:05,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:05,309 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:05,428 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:06,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:07,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:08,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:09,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:10,246 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:10,752 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:11,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:12,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:12,283 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:59:12,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:59:12,673 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:59:13,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:14,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:15,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:15,775 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:16,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:17,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:18,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:19,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:20,247 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:21,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:21,753 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:22,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:23,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:24,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:25,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:26,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:27,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:27,673 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:59:27,674 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:59:27,795 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:28,248 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:29,249 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:30,297 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:31,274 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:31,716 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 14:59:31,716 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 with policy now |
| 2025-01-17 14:59:31,720 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:59:31,721 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:59:31,721 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:59:31,721 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:59:31,998 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/1ah9u18q-media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 |
| 2025-01-17 14:59:32,281 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:32,440 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:59:32,440 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:32,441 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 |
| 2025-01-17 14:59:32,441 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 14:59:32,753 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/9n8n3cul-media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 |
| 2025-01-17 14:59:32,807 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:33,287 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:33,441 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:34,290 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:34,441 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:35,311 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:36,295 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:37,298 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:38,296 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:38,443 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:38,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:39,300 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:40,297 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:40,444 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:41,313 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:42,284 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 14:59:42,329 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:42,444 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:59:42,641 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 14:59:42,641 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:59:42,642 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:59:42,697 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:59:42,697 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:59:43,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:43,445 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 14:59:43,445 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 14:59:44,322 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:44,445 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:44,477 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:45,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:46,360 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:46,446 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:47,325 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:48,325 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:48,447 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:49,325 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:49,797 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:50,325 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:50,448 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 14:59:51,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:52,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:53,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:54,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:55,326 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:55,765 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:59:56,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:57,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:57,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:59:57,679 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:59:58,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:59:59,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:00,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:00,800 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:01,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:02,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:03,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:04,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:05,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:06,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:06,766 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:07,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:08,327 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:09,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:10,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:11,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:11,766 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:12,284 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:00:12,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:12,679 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:00:12,680 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:00:13,328 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:14,345 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:15,348 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:15,621 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:00:15,621 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 with policy now |
| 2025-01-17 15:00:15,643 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:00:15,644 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:00:15,644 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:00:15,644 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:00:15,886 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/553ul7rh-media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 |
| 2025-01-17 15:00:16,345 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:16,459 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:00:16,459 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:16,459 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 |
| 2025-01-17 15:00:16,459 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 15:00:16,753 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/51414zxw-media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 |
| 2025-01-17 15:00:17,347 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:17,459 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:17,734 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:18,350 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:18,460 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:19,350 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:20,353 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:21,376 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:22,363 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:22,461 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:22,766 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:23,392 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:23,688 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:00:23,688 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:00:23,693 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:00:24,391 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:24,462 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:00:24,462 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:24,463 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:00:25,425 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:26,400 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:26,463 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:27,416 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:27,694 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:00:27,694 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:00:27,846 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:28,423 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:29,421 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:30,419 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:30,465 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:31,421 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:32,436 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:33,463 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:33,814 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:34,456 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:35,466 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:36,469 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:37,486 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:38,456 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:38,982 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:39,456 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:40,456 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:40,469 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:00:41,456 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:42,285 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:00:42,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:42,684 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:00:42,684 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:00:43,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:44,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:44,964 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:45,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:46,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:47,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:48,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:49,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:50,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:50,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:51,457 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:52,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:53,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:54,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:55,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:55,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:00:56,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:57,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:57,692 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:00:57,692 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:00:58,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:00:59,459 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:00,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:00,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:01,458 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:02,459 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:03,459 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:04,459 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:05,480 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:06,046 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:06,213 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:01:06,213 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 with policy now |
| 2025-01-17 15:01:06,235 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:01:06,236 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:01:06,236 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:01:06,236 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:01:06,473 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:06,480 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:01:06,480 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:06,480 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 |
| 2025-01-17 15:01:06,481 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 15:01:06,483 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/5skhsm87-media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 |
| 2025-01-17 15:01:06,756 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/mctt4p8j-media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 |
| 2025-01-17 15:01:07,480 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:07,493 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:08,481 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:08,501 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:09,583 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:10,550 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:11,553 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:11,981 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:12,285 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:01:12,483 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:12,546 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:12,734 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:01:12,735 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:01:13,530 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:14,484 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:01:14,539 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:14,755 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:01:14,755 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:01:14,755 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:01:14,780 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/g83t1x47-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:01:15,484 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:01:15,577 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:16,484 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:16,563 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:17,245 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:17,606 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:18,485 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:18,559 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:19,559 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:20,486 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:20,559 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:21,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:22,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:22,762 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:23,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:24,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:25,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:26,489 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:26,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:27,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:27,691 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:01:27,691 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:01:27,812 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:28,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:29,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:30,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:31,560 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:32,561 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:32,824 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:33,561 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:34,561 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:35,561 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:36,561 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:37,562 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:37,824 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:38,562 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:39,562 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:40,562 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:41,562 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:42,285 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:01:42,563 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:42,692 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:01:42,692 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:01:42,838 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:43,563 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:44,563 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:45,576 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:46,498 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:46,583 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:47,580 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:47,767 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:01:47,767 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 with policy now |
| 2025-01-17 15:01:47,793 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:01:47,793 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:01:47,793 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:01:47,793 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:01:47,840 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:48,035 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/lqhk7obw-media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 |
| 2025-01-17 15:01:48,499 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:01:48,499 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:48,499 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 |
| 2025-01-17 15:01:48,499 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 15:01:48,624 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:48,794 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/3avpp5qc-media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 |
| 2025-01-17 15:01:49,499 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:49,612 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:50,500 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:50,595 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:51,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:52,623 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:52,897 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:53,618 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:54,501 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:54,599 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:55,639 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:56,502 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:01:56,641 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:57,615 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:57,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:01:57,756 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:01:57,957 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:01:58,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:01:59,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:00,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:00,811 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:02:00,811 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:02:00,812 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:02:01,504 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:02:01,505 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:02:01,652 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:02,505 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:02,626 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:03,662 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:03,883 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:04,506 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:04,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:05,627 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:06,507 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:06,643 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:07,671 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:08,508 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:08,662 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:09,681 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:09,873 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:10,655 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:11,645 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:12,286 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:02:12,509 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:12,645 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:12,712 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:02:12,713 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:02:13,645 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:14,510 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:14,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:15,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:15,863 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:16,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:17,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:18,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:19,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:20,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:21,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:21,860 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:22,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:23,646 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:24,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:25,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:26,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:26,860 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:27,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:27,713 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:02:27,713 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:02:28,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:29,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:30,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:31,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:31,861 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:32,647 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:33,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:34,648 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:35,651 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:36,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:37,650 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:37,860 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:38,657 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:39,426 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:02:39,426 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 with policy now |
| 2025-01-17 15:02:39,447 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:02:39,448 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:02:39,448 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:02:39,448 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:02:39,521 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:02:39,522 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 |
| 2025-01-17 15:02:39,522 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 15:02:39,659 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:39,699 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/qxfe0zj5-media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 |
| 2025-01-17 15:02:39,976 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/sxl2cva2-media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 |
| 2025-01-17 15:02:40,522 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:40,675 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:41,522 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:41,703 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:42,286 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:02:42,523 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:42,682 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:42,772 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:02:42,773 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:02:42,912 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:43,711 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:44,524 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:44,687 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:45,729 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:46,525 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:46,678 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:47,685 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:47,868 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:02:47,869 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:02:47,869 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:02:47,937 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:48,526 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:02:48,526 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:48,526 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:02:48,749 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:49,740 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:50,527 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:50,715 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:51,714 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:52,719 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:53,763 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:53,909 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:02:54,528 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:02:54,730 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:55,764 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:56,741 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:57,735 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:57,764 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:02:57,765 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:02:58,742 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:59,761 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:02:59,904 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:00,766 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:01,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:02,793 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:03,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:04,533 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:04,753 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:05,151 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:05,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:06,535 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:06,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:07,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:08,536 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:08,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:09,754 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:10,537 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:10,745 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:10,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:11,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:12,287 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:03:12,538 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:12,743 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:03:12,743 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:03:12,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:13,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:14,539 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:14,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:15,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:15,985 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:16,540 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:16,755 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:17,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:18,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:19,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:20,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:21,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:21,891 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:22,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:23,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:24,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:25,756 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:26,544 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:26,789 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:27,743 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:03:27,743 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:03:27,789 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:27,917 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:28,545 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:28,757 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:29,757 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:30,798 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:31,602 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:03:31,602 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 with policy now |
| 2025-01-17 15:03:31,640 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:03:31,640 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:03:31,640 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:31,641 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:31,780 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:31,872 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/x46k3m8c-media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 |
| 2025-01-17 15:03:32,547 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:03:32,547 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:32,547 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 |
| 2025-01-17 15:03:32,548 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval |
| 2025-01-17 15:03:32,799 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:32,871 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/jl6hd4vq-media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 |
| 2025-01-17 15:03:33,548 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:33,791 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:33,913 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:34,548 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:34,840 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:35,802 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:36,839 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:37,840 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:38,550 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:38,831 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:38,913 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:39,550 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:03:39,834 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:39,932 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:03:39,932 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:39,933 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:40,551 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:03:40,551 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:40,551 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:03:40,831 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:40,921 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 15:03:40,921 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 15:03:41,139 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:03:41,551 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:03:41,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json |
| 2025-01-17 15:03:41,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media |
| 2025-01-17 15:03:41,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval |
| 2025-01-17 15:03:41,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table |
| 2025-01-17 15:03:41,558 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 15:03:41,570 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 15:03:41,635 INFO wandb-upload_2:2517552 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpf7i9yzmj |
| 2025-01-17 15:03:41,793 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/thkplcdy-media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json |
| 2025-01-17 15:03:41,802 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:41,855 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/qtky6ri1-events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:03:42,076 INFO SenderThread:2517552 [sender.py:send_request_log_artifact():1525] logged artifact run-up07mbix-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTEzMjcwMg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTU2Mjc2NA==', 'latestArtifact': None}} |
| 2025-01-17 15:03:42,076 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:03:42,076 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json with policy now |
| 2025-01-17 15:03:42,076 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:03:42,076 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,077 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,077 DEBUG SenderThread:2517552 [sender.py:send():379] send: telemetry |
| 2025-01-17 15:03:42,077 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,081 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,081 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,081 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,082 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,082 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,082 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,083 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,083 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,083 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,083 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,084 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,084 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,084 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,085 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,085 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,085 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,085 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,086 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,086 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,086 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,087 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,087 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 15:03:42,343 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/ulxn4kk7-media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json |
| 2025-01-17 15:03:42,473 INFO wandb-upload_3:2517552 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppfo4cq0c |
| 2025-01-17 15:03:42,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:03:42,553 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:42,553 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json |
| 2025-01-17 15:03:42,553 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval |
| 2025-01-17 15:03:42,743 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:03:42,802 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:42,904 INFO SenderThread:2517552 [sender.py:send_request_log_artifact():1525] logged artifact run-up07mbix-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTEzMjcyNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTU2Mjc2Nw==', 'latestArtifact': None}} |
| 2025-01-17 15:03:42,904 DEBUG SenderThread:2517552 [sender.py:send():379] send: files |
| 2025-01-17 15:03:42,904 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json with policy now |
| 2025-01-17 15:03:42,904 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:03:42,904 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:03:42,905 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:03:42,905 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:03:42,905 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:03:43,158 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/6l91m2mj-media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json |
| 2025-01-17 15:03:43,428 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /tmp/tmp7sbm32f9wandb/g6lnbs6v-media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json |
| 2025-01-17 15:03:43,552 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:03:43,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:43,972 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:44,553 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:03:44,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:45,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:46,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:47,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:48,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:49,803 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:49,894 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:50,804 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:51,804 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:52,804 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:53,804 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:54,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:54,894 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:03:55,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:56,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:57,743 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 15:03:57,743 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 15:03:57,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:58,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:59,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:03:59,898 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:04:00,561 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/config.yaml |
| 2025-01-17 15:04:00,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:04:01,805 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:04:02,405 DEBUG SenderThread:2517552 [sender.py:send():379] send: exit |
| 2025-01-17 15:04:02,405 INFO SenderThread:2517552 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-17 15:04:02,406 INFO SenderThread:2517552 [sender.py:send_exit():588] handling runtime: 980 |
| 2025-01-17 15:04:02,406 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:04:02,406 INFO SenderThread:2517552 [sender.py:send_exit():594] send defer |
| 2025-01-17 15:04:02,406 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:02,406 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-17 15:04:02,406 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:02,406 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-17 15:04:02,406 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-17 15:04:02,407 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:02,407 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-17 15:04:02,407 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:02,407 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-17 15:04:02,407 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-17 15:04:02,407 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:02,407 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-17 15:04:02,407 INFO HandlerThread:2517552 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-17 15:04:02,407 INFO HandlerThread:2517552 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-17 15:04:02,407 DEBUG SystemMonitor:2517552 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-17 15:04:02,408 DEBUG SystemMonitor:2517552 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-17 15:04:02,408 INFO HandlerThread:2517552 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-17 15:04:02,458 INFO HandlerThread:2517552 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-17 15:04:02,459 INFO HandlerThread:2517552 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-17 15:04:02,459 INFO HandlerThread:2517552 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-17 15:04:02,459 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:02,459 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-17 15:04:02,459 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-17 15:04:02,459 DEBUG SenderThread:2517552 [sender.py:send():379] send: stats |
| 2025-01-17 15:04:02,460 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:02,460 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-17 15:04:02,460 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:02,460 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-17 15:04:02,460 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-17 15:04:02,460 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:02,460 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-17 15:04:02,562 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:04:04,564 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:04:08,941 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 15:04:08,941 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:04:08,941 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-17 15:04:08,942 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:08,942 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-17 15:04:08,942 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-17 15:04:08,942 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 15:04:08,942 DEBUG SenderThread:2517552 [sender.py:send():379] send: history |
| 2025-01-17 15:04:08,943 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 15:04:08,943 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:08,943 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-17 15:04:08,944 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:04:08,944 DEBUG SenderThread:2517552 [sender.py:send():379] send: summary |
| 2025-01-17 15:04:08,948 INFO SenderThread:2517552 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 15:04:08,948 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:08,948 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-17 15:04:08,948 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-17 15:04:08,948 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:08,948 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-17 15:04:08,948 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:08,948 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-17 15:04:08,948 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-17 15:04:08,948 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 15:04:08,949 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:08,949 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-17 15:04:08,949 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:08,949 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-17 15:04:09,407 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 15:04:09,566 INFO Thread-12 :2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:04:11,093 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-17 15:04:11,093 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 15:04:11,094 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:11,094 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-17 15:04:11,094 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:11,094 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-17 15:04:11,094 INFO SenderThread:2517552 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-17 15:04:11,097 INFO SenderThread:2517552 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-17 15:04:11,099 INFO SenderThread:2517552 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-17 15:04:11,113 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-17 15:04:11,113 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:11,114 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-17 15:04:11,114 DEBUG SenderThread:2517552 [sender.py:send():379] send: artifact |
| 2025-01-17 15:04:11,314 INFO SenderThread:2517552 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ0OTEwNDY1OA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ0OTEwNDY1OA==', 'versionIndex': 442}}} |
| 2025-01-17 15:04:11,314 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:11,314 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-17 15:04:11,314 INFO SenderThread:2517552 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-17 15:04:11,407 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 15:04:11,567 INFO SenderThread:2517552 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:04:11,567 INFO SenderThread:2517552 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/config.yaml config.yaml |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json wandb-summary.json |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/requirements.txt requirements.txt |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 events.out.tfevents.1737143262.leonmkim-ROG-Strix-G15CS-G15CS.2517388.0 |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log output.log |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/diff.patch diff.patch |
| 2025-01-17 15:04:11,568 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-17 15:04:11,574 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 media/videos/eval/7_eval_7_baeae50ab773d27ae44b.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 media/videos/eval/10_eval_10_268ff9e2e53b41188aa0.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 media/videos/eval/16_eval_16_d19b3abe759fefca6827.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 media/videos/eval/0_eval_0_1819f930b352d4eea54c.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 media/videos/eval/6_eval_6_a92f6e4d8653d4439383.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 media/videos/eval/14_eval_14_bfd6e56a3506bb346792.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 media/videos/eval/17_eval_17_9039e216f093cde1d3f4.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 media/videos/eval/12_eval_12_79b7fe799da0fe5e29e5.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 media/videos/eval/1_eval_1_093e197b5aca9677627b.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 media/videos/eval/15_eval_15_9e02939fb08b9aeb13e6.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 media/videos/eval/4_eval_4_916d91435e19178c30f6.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 media/videos/eval/5_eval_5_abbb012eb02e7137181f.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 media/videos/eval/3_eval_3_ccfd7c2808a5399343db.mp4 |
| 2025-01-17 15:04:11,575 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 media/videos/eval/13_eval_13_72586beaede9059dffaa.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 media/videos/eval/8_eval_8_f6383c0f370fe43d88dd.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 media/videos/eval/18_eval_18_d73f4283df1c3c229b04.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 media/videos/eval/9_eval_9_a200f16787e6024f6581.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 media/videos/eval/19_eval_19_1265d28d813382198c39.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 media/videos/eval/11_eval_11_30dc16b787631229f366.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 media/videos/eval/2_eval_2_ee6ae32e8cb4b17611f2.mp4 |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json media/table/eval/total_success_rate_ci_21_37909e05a422137136b7.table.json |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json media/table/eval/success_rate_ci_20_fc42d684313cd17be1f8.table.json |
| 2025-01-17 15:04:11,576 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-17 15:04:11,576 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 15:04:11,576 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:11,577 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-17 15:04:11,577 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:11,577 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-17 15:04:11,577 INFO SenderThread:2517552 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 15:04:11,763 INFO wandb-upload_2:2517552 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/wandb-summary.json |
| 2025-01-17 15:04:11,792 INFO wandb-upload_3:2517552 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/requirements.txt |
| 2025-01-17 15:04:11,805 INFO wandb-upload_0:2517552 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/config.yaml |
| 2025-01-17 15:04:11,814 INFO wandb-upload_1:2517552 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/files/output.log |
| 2025-01-17 15:04:12,015 INFO Thread-11 (_thread_body):2517552 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-17 15:04:12,015 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:12,015 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-17 15:04:12,015 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:12,015 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-17 15:04:12,015 INFO SenderThread:2517552 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-17 15:04:12,019 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-17 15:04:12,019 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:12,019 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-17 15:04:12,019 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:12,019 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-17 15:04:12,019 INFO SenderThread:2517552 [file_stream.py:finish():601] file stream finish called |
| 2025-01-17 15:04:12,217 INFO SenderThread:2517552 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-17 15:04:12,217 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-17 15:04:12,217 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:12,217 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-17 15:04:12,217 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:12,217 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-17 15:04:12,217 INFO SenderThread:2517552 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-17 15:04:12,218 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 15:04:12,218 INFO HandlerThread:2517552 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-17 15:04:12,218 DEBUG SenderThread:2517552 [sender.py:send():379] send: final |
| 2025-01-17 15:04:12,218 DEBUG SenderThread:2517552 [sender.py:send():379] send: footer |
| 2025-01-17 15:04:12,218 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 15:04:12,218 INFO SenderThread:2517552 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-17 15:04:12,218 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 15:04:12,219 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 15:04:12,219 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-17 15:04:12,219 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 15:04:12,219 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: server_info |
| 2025-01-17 15:04:12,219 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-17 15:04:12,221 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-17 15:04:12,222 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 15:04:12,282 DEBUG SenderThread:2517552 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 15:04:12,282 INFO MainThread:2517552 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-17 15:04:12,283 INFO MainThread:2517552 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-17 15:04:12,283 INFO MainThread:2517552 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-17 15:04:12,283 DEBUG HandlerThread:2517552 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-17 15:04:12,283 INFO HandlerThread:2517552 [handler.py:finish():882] shutting down handler |
| 2025-01-17 15:04:13,220 INFO WriterThread:2517552 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/up07mbix/wandb/run-20250117_144741-up07mbix/run-up07mbix.wandb |
| 2025-01-17 15:04:13,282 INFO SenderThread:2517552 [sender.py:finish():1615] shutting down sender |
| 2025-01-17 15:04:13,282 INFO SenderThread:2517552 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 15:04:13,282 INFO SenderThread:2517552 [file_pusher.py:join():175] waiting for file pusher |
|
|
