| 2025-01-17 14:29:32,140 INFO StreamThr :2514330 [internal.py:wandb_internal():85] W&B internal server running at pid: 2514330, started at: 2025-01-17 14:29:32.140380 |
| 2025-01-17 14:29:32,141 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status |
| 2025-01-17 14:29:32,157 INFO WriterThread:2514330 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/run-odxnffav.wandb |
| 2025-01-17 14:29:32,158 DEBUG SenderThread:2514330 [sender.py:send():379] send: header |
| 2025-01-17 14:29:32,169 DEBUG SenderThread:2514330 [sender.py:send():379] send: run |
| 2025-01-17 14:29:32,582 INFO SenderThread:2514330 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files |
| 2025-01-17 14:29:32,583 INFO SenderThread:2514330 [sender.py:_start_run_threads():1188] run started: odxnffav with start time 1737142172.140455 |
| 2025-01-17 14:29:32,600 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-17 14:29:32,600 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: check_version |
| 2025-01-17 14:29:32,642 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-17 14:29:32,647 DEBUG HandlerThread:2514330 [system_info.py:__init__():26] System info init |
| 2025-01-17 14:29:32,647 DEBUG HandlerThread:2514330 [system_info.py:__init__():41] System info init done |
| 2025-01-17 14:29:32,647 INFO HandlerThread:2514330 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-17 14:29:32,647 INFO SystemMonitor:2514330 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-17 14:29:32,647 INFO HandlerThread:2514330 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-17 14:29:32,648 INFO SystemMonitor:2514330 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-17 14:29:32,648 INFO SystemMonitor:2514330 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-17 14:29:32,649 INFO SystemMonitor:2514330 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-17 14:29:32,650 INFO SystemMonitor:2514330 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-17 14:29:32,650 INFO SystemMonitor:2514330 [interfaces.py:start():188] Started network monitoring |
| 2025-01-17 14:29:32,661 DEBUG HandlerThread:2514330 [system_info.py:probe():152] Probing system |
| 2025-01-17 14:29:32,663 DEBUG HandlerThread:2514330 [system_info.py:_probe_git():137] Probing git |
| 2025-01-17 14:29:32,667 DEBUG HandlerThread:2514330 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-17 14:29:32,667 DEBUG HandlerThread:2514330 [system_info.py:probe():200] Probing system done |
| 2025-01-17 14:29:32,667 DEBUG HandlerThread:2514330 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-17T19:29:32.661542', 'startedAt': '2025-01-17T19:29:32.128161', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3563.8643749999997, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.002, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4501.083, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 374.1404724121094}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-17 14:29:32,667 INFO HandlerThread:2514330 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-17 14:29:32,668 INFO HandlerThread:2514330 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-17 14:29:32,668 DEBUG HandlerThread:2514330 [system_info.py:_save_code():44] Saving code |
| 2025-01-17 14:29:32,673 DEBUG HandlerThread:2514330 [system_info.py:_save_code():67] Saving code done |
| 2025-01-17 14:29:32,673 DEBUG HandlerThread:2514330 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-17 14:29:32,768 DEBUG HandlerThread:2514330 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-17 14:29:32,769 INFO HandlerThread:2514330 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-17 14:29:32,772 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:29:32,772 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-17 14:29:32,772 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-17 14:29:32,772 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-17 14:29:32,773 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-17 14:29:33,000 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-17 14:29:33,001 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-17 14:29:33,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:29:33,001 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:29:33,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:33,005 INFO HandlerThread:2514330 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/tb" |
| save: true |
| } |
| control { |
| relay_id: "4d9c6c5a7c254e1f9a8ed9d676a8df15" |
| } |
| _info { |
| stream_id: "odxnffav" |
| } |
|
|
| 2025-01-17 14:29:33,005 DEBUG HandlerThread:2514330 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-17 14:29:33,020 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/bb0d3sle-wandb-metadata.json |
| 2025-01-17 14:29:33,058 DEBUG SenderThread:2514330 [sender.py:send():379] send: telemetry |
| 2025-01-17 14:29:33,058 DEBUG SenderThread:2514330 [sender.py:send():379] send: tbrecord |
| 2025-01-17 14:29:33,059 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:29:33,059 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 with policy live |
| 2025-01-17 14:29:33,067 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/yormr7cw-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 14:29:33,076 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/jftrgxs5-code/FISH/eval_robot.py |
| 2025-01-17 14:29:33,079 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/dtth7ds4-diff.patch |
| 2025-01-17 14:29:33,266 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/bpcho8yu-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:29:33,584 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 14:29:33,584 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/diff.patch |
| 2025-01-17 14:29:33,584 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/requirements.txt |
| 2025-01-17 14:29:33,584 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-metadata.json |
| 2025-01-17 14:29:33,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:29:33,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/code/FISH/eval_robot.py |
| 2025-01-17 14:29:33,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:29:33,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/code |
| 2025-01-17 14:29:33,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/code/FISH |
| 2025-01-17 14:29:34,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:35,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:35,583 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:29:36,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:37,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:38,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:29:38,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:39,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:40,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:40,915 DEBUG SenderThread:2514330 [sender.py:send():379] send: config |
| 2025-01-17 14:29:41,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:42,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:43,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:43,585 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:29:44,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:29:44,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:45,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:46,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:47,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:47,587 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:29:48,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:29:48,001 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:29:48,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:49,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:49,445 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:29:49,587 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:29:50,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:51,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:52,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:53,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:54,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:55,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:29:55,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:56,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:57,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:58,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:29:59,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:00,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:00,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:01,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:02,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:03,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:30:03,001 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:30:03,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:04,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:05,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:05,160 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:05,592 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/config.yaml |
| 2025-01-17 14:30:06,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:07,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:08,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:09,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:10,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:11,002 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:11,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:12,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:13,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:13,594 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:14,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:15,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:15,595 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:16,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:16,578 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:17,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:17,595 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:18,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:30:18,002 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:30:18,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:19,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:19,596 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:20,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:21,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:22,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:22,069 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:23,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:24,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:25,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:26,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:27,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:28,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:28,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:29,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:30,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:31,010 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:32,010 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:32,651 DEBUG SystemMonitor:2514330 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-17 14:30:32,651 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:30:33,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:30:33,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:30:33,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:33,140 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:34,010 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:35,010 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:36,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:37,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:38,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:39,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:39,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:40,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:41,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:42,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:43,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:44,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:44,313 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:44,513 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:30:44,513 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 with policy now |
| 2025-01-17 14:30:44,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:30:44,514 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:30:44,514 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:30:44,516 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:30:44,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 |
| 2025-01-17 14:30:44,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:30:44,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:30:44,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos |
| 2025-01-17 14:30:44,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media |
| 2025-01-17 14:30:44,750 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/5yktu7tk-media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 |
| 2025-01-17 14:30:45,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:45,021 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/fb00d24w-media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 |
| 2025-01-17 14:30:45,603 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:46,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:47,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:48,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:30:48,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:30:48,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:49,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:49,605 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:50,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:50,137 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:51,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:52,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:53,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:53,606 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:30:54,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:55,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:56,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:30:56,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:57,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:58,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:30:59,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:00,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:01,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:01,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:02,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:02,652 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:31:03,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:31:03,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:31:03,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:04,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:05,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:06,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:06,145 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:07,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:07,612 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:31:07,871 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/1xkuaz4f-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:31:08,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:09,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:09,613 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:10,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:11,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:11,614 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:11,849 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:12,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:13,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:13,615 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:14,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:15,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:16,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:17,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:17,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:18,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:31:18,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:31:18,049 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:19,016 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:20,016 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:21,016 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:22,016 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:22,081 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:23,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:24,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:25,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:25,618 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:26,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:27,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:27,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:28,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:29,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:30,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:31,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:31,620 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:32,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:32,652 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:31:32,653 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:33,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:31:33,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:31:33,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:34,020 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:35,021 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:36,021 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:37,021 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:38,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:38,021 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:38,618 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:31:38,618 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 with policy now |
| 2025-01-17 14:31:38,619 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:31:38,619 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:31:38,619 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:31:38,621 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:31:38,623 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:31:38,623 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 |
| 2025-01-17 14:31:38,623 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:31:38,870 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/9dxwxzrm-media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 |
| 2025-01-17 14:31:39,022 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:39,148 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/b91yv1ob-media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 |
| 2025-01-17 14:31:39,623 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:40,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:41,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:42,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:43,010 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:43,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:43,624 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:44,026 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:45,061 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:46,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:47,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:47,626 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:48,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:31:48,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:31:48,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:48,145 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:48,438 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:31:48,438 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:31:48,438 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:31:48,626 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:31:48,626 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:31:49,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:49,626 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:50,028 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:51,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:51,627 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:52,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:53,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:53,628 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:31:54,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:31:54,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:55,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:56,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:57,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:58,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:31:59,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:00,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:00,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:01,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:02,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:02,653 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:32:03,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:32:03,004 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:32:03,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:04,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:05,088 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:05,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:06,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:07,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:08,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:09,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:10,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:11,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:11,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:12,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:13,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:14,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:15,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:16,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:16,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:17,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:18,004 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:32:18,005 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:32:18,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:19,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:20,094 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:20,358 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:32:20,358 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 with policy now |
| 2025-01-17 14:32:20,359 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:32:20,360 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:32:20,360 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:32:20,361 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:32:20,598 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/zetewlf8-media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 |
| 2025-01-17 14:32:20,637 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:32:20,637 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 |
| 2025-01-17 14:32:20,637 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:32:20,921 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/uhp1oif3-media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 |
| 2025-01-17 14:32:21,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:21,387 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:21,637 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:22,101 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:22,638 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:23,104 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:24,105 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:25,107 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:25,639 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:26,104 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:26,881 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:27,102 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:28,109 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:29,106 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:29,640 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:30,107 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:31,106 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:31,275 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:32:31,275 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:32:31,276 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:32:31,641 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:32:31,642 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:32:31,934 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/5u52yykk-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:32:32,020 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:32,117 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:32,653 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:32:33,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:32:33,007 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:32:33,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:33,642 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:34,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:35,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:36,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:37,091 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:37,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:37,643 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:32:38,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:39,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:40,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:41,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:42,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:43,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:43,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:44,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:45,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:46,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:47,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:48,006 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:32:48,006 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:32:48,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:48,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:49,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:50,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:51,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:52,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:53,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:54,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:54,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:55,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:56,112 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:57,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:58,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:32:59,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:32:59,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:00,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:01,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:02,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:02,654 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:33:03,005 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:33:03,006 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:33:03,118 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:04,122 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:04,139 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:04,211 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:33:04,211 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 with policy now |
| 2025-01-17 14:33:04,214 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:33:04,215 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:33:04,215 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:33:04,215 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:33:04,444 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/mqc8hnwd-media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 |
| 2025-01-17 14:33:04,653 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:33:04,653 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 |
| 2025-01-17 14:33:04,653 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:33:04,939 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/y1i3iiru-media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 |
| 2025-01-17 14:33:05,120 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:05,653 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:06,122 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:06,653 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:07,123 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:08,129 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:09,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:09,654 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:10,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:10,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:11,125 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:11,655 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:12,127 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:13,129 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:13,327 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:33:13,327 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:33:13,328 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:33:13,656 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:33:13,656 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:33:14,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:15,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:15,185 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:15,657 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:16,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:17,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:18,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:33:18,013 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:33:18,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:19,130 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:19,658 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:20,130 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:20,213 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:21,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:21,659 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:22,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:23,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:23,659 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:24,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:25,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:25,661 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:25,805 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:26,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:27,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:28,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:29,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:29,662 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:30,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:31,018 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:31,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:32,132 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:32,654 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:33:33,007 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:33:33,007 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:33:33,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:34,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:35,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:36,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:36,154 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:37,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:38,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:39,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:40,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:41,134 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:42,019 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:42,135 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:43,135 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:44,135 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:45,135 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:46,146 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:46,522 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:33:46,522 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 with policy now |
| 2025-01-17 14:33:46,523 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:33:46,523 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:33:46,523 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:33:46,525 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:33:46,668 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:33:46,668 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 |
| 2025-01-17 14:33:46,668 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:33:46,772 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/e6lkw693-media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 |
| 2025-01-17 14:33:47,039 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/vsihdu4v-media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 |
| 2025-01-17 14:33:47,143 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:47,575 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:47,668 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:48,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:33:48,012 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:33:48,147 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:49,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:50,146 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:51,146 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:51,670 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:52,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:53,030 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:53,145 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:54,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:55,155 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:55,671 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:56,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:56,382 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:33:56,382 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:33:56,383 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:33:56,671 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:33:56,672 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:33:56,996 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/q7yso7ye-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:33:57,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:57,672 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:33:58,155 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:59,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:33:59,151 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:33:59,672 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:00,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:01,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:01,673 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:02,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:02,655 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:34:03,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:34:03,009 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:34:03,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:03,674 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:04,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:04,154 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:05,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:06,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:07,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:08,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:09,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:10,022 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:10,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:11,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:12,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:13,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:14,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:15,150 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:16,022 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:16,151 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:17,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:18,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:34:18,009 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:34:18,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:19,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:20,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:21,122 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:21,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:22,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:23,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:24,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:25,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:26,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:26,807 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:27,155 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:27,683 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:27,892 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:34:27,893 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 with policy now |
| 2025-01-17 14:34:27,895 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:34:27,896 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:34:27,896 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:34:27,896 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:34:28,126 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/kzp17l1y-media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 |
| 2025-01-17 14:34:28,156 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:28,683 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:34:28,683 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 |
| 2025-01-17 14:34:28,683 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:34:28,964 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/lvqu7e2s-media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 |
| 2025-01-17 14:34:29,165 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:29,683 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:30,179 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:31,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:32,175 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:32,613 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:32,655 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:34:33,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:34:33,014 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:34:33,176 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:33,685 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:34,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:35,193 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:35,685 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:36,186 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:37,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:38,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:38,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:39,195 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:40,199 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:40,687 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:34:41,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:41,439 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:34:41,439 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:34:41,440 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:34:41,688 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:34:41,688 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:41,688 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:34:42,193 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:43,037 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:43,196 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:43,688 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:44,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:45,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:45,689 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:46,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:47,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:48,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:34:48,009 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:34:48,168 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:48,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:49,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:50,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:51,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:52,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:53,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:53,692 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:54,023 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:54,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:55,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:55,693 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:56,189 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:57,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:57,694 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:34:58,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:59,024 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:34:59,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:34:59,694 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:00,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:01,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:01,695 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:02,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:02,655 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:35:03,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:35:03,010 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:35:03,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:03,696 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:04,111 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:04,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:05,190 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:06,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:07,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:07,697 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:08,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:09,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:09,698 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:10,001 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:10,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:11,191 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:11,699 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:12,194 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:12,891 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:35:12,891 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 with policy now |
| 2025-01-17 14:35:12,900 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:35:12,901 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:35:12,901 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:35:12,901 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:35:13,121 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/hch8vyvc-media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 |
| 2025-01-17 14:35:13,193 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:13,700 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:13,700 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:35:13,700 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 |
| 2025-01-17 14:35:13,700 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:35:13,984 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/duthr6v8-media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 |
| 2025-01-17 14:35:14,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:14,700 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:15,025 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:15,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:16,201 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:17,207 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:17,701 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:18,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:35:18,016 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:35:18,207 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:19,208 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:20,207 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:20,458 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:21,212 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:21,703 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:22,209 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:23,209 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:23,492 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:35:23,492 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:35:23,493 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:35:23,704 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:35:23,704 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:35:23,957 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/fa8ohva5-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:35:24,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:25,217 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:25,705 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:26,153 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:26,234 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:27,230 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:27,705 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:28,227 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:29,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:30,226 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:31,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:31,318 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:31,707 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:32,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:32,656 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:35:33,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:35:33,011 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:35:33,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:34,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:35,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:35,708 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:36,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:37,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:37,219 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:38,219 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:39,219 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:40,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:41,219 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:41,711 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:35:42,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:42,497 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:43,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:44,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:45,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:46,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:47,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:48,011 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:35:48,011 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:35:48,167 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:48,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:49,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:50,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:51,220 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:52,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:53,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:54,035 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:54,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:55,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:56,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:57,221 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:58,239 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:58,309 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:35:58,309 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:35:58,310 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 with policy now |
| 2025-01-17 14:35:58,310 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:35:58,310 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:35:58,311 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:35:58,554 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/u9aybhrw-media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 |
| 2025-01-17 14:35:58,717 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:35:58,717 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 |
| 2025-01-17 14:35:58,717 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:35:58,998 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/lwdxb3bn-media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 |
| 2025-01-17 14:35:59,238 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:35:59,334 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:35:59,717 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:00,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:00,718 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:01,240 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:02,250 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:02,656 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:36:03,026 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:36:03,026 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:36:03,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:03,719 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:04,249 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:04,882 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:05,246 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:05,719 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:06,250 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:07,262 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:08,262 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:09,254 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:09,721 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:36:10,263 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:10,551 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:36:10,551 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:36:10,551 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:10,551 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:36:10,721 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:36:11,265 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:11,722 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:12,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:13,259 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:14,252 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:15,254 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:15,723 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:15,957 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:16,253 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:17,253 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:18,013 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:36:18,013 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:36:18,254 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:19,254 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:20,255 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:21,163 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:21,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:22,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:23,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:24,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:25,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:26,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:27,042 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:27,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:28,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:29,256 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:30,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:31,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:32,042 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:32,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:32,657 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:36:33,014 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:36:33,014 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:36:33,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:34,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:35,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:36,257 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:37,133 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:37,258 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:38,258 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:39,258 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:40,258 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:41,268 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:41,733 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:42,142 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:42,269 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:42,292 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:36:42,292 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 with policy now |
| 2025-01-17 14:36:42,333 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:36:42,333 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:36:42,334 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:36:42,334 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:36:42,518 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/b2xifp9z-media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 |
| 2025-01-17 14:36:42,734 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:42,734 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:36:42,734 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 |
| 2025-01-17 14:36:42,735 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:36:43,002 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/7tx1w5el-media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 |
| 2025-01-17 14:36:43,272 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:43,734 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:44,272 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:44,734 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:45,269 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:46,272 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:47,278 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:47,735 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:48,023 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:36:48,023 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:36:48,081 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:48,275 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:49,271 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:49,736 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:50,278 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:51,278 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:51,600 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:36:51,601 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:36:51,601 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:36:51,737 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:36:51,737 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:36:52,001 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/jgu33evp-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:36:52,291 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:53,296 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:53,738 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:54,072 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:36:54,319 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:55,292 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:56,278 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:57,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:57,739 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:36:58,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:59,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:36:59,741 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:00,062 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:00,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:01,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:02,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:02,657 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:37:03,018 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:37:03,019 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:37:03,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:04,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:05,182 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:05,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:06,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:07,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:08,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:09,279 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:10,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:11,062 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:11,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:12,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:13,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:14,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:15,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:16,063 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:16,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:17,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:18,019 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:37:18,019 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:37:18,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:19,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:20,280 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:21,143 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:21,281 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:22,281 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:23,309 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:23,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:23,838 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:37:23,838 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 with policy now |
| 2025-01-17 14:37:23,848 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:37:23,849 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:37:23,849 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:37:23,849 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:37:24,069 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/fdqkbph8-media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 |
| 2025-01-17 14:37:24,288 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:24,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:24,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:37:24,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 |
| 2025-01-17 14:37:24,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:37:25,051 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/cz2a2lup-media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 |
| 2025-01-17 14:37:25,301 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:25,750 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:26,306 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:26,931 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:27,303 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:28,312 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:29,313 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:29,752 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:30,315 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:31,320 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:31,753 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:32,338 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:32,658 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:37:32,659 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:33,040 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:37:33,040 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:37:33,321 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:34,337 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:35,339 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:35,650 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:37:35,650 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:37:35,651 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:37:35,754 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:37:35,754 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:37:36,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:37,345 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:37,755 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:37,769 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:38,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:39,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:40,339 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:41,345 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:41,756 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:42,359 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:42,833 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:43,363 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:44,364 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:45,370 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:46,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:47,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:47,759 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:37:47,924 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:48,028 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:37:48,029 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:37:48,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:49,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:50,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:51,353 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:52,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:53,078 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:53,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:54,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:55,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:56,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:57,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:58,078 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:37:58,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:37:59,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:00,354 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:01,355 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:02,373 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:02,658 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:38:03,049 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:38:03,049 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:38:03,162 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:03,381 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:03,629 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:38:03,629 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 with policy now |
| 2025-01-17 14:38:03,641 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:38:03,641 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:38:03,642 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:38:03,642 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:38:03,765 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:03,765 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:38:03,765 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 |
| 2025-01-17 14:38:03,766 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:38:03,850 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/z9tzmpyw-media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 |
| 2025-01-17 14:38:04,116 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/njlcov0v-media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 |
| 2025-01-17 14:38:04,390 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:04,765 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:05,382 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:05,766 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:06,383 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:07,409 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:08,408 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:09,103 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:09,446 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:09,767 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:10,409 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:11,411 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:11,768 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:12,423 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:13,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:14,437 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:14,459 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:14,702 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:38:14,702 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:38:14,702 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:38:14,769 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:38:14,770 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:38:15,071 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/wbsxlg8g-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:38:15,447 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:15,770 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:16,412 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:17,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:17,771 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:18,075 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:38:18,075 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:38:18,413 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:19,431 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:19,707 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:19,771 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:20,435 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:21,438 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:22,438 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:23,439 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:24,438 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:24,733 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:25,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:25,774 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:26,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:27,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:28,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:29,426 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:30,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:30,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:31,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:32,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:32,659 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:38:33,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:38:33,045 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:38:33,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:34,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:35,126 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:35,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:36,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:37,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:38,427 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:39,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:40,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:41,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:41,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:42,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:43,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:44,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:45,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:46,090 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:46,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:47,429 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:48,045 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:38:48,045 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:38:48,429 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:49,429 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:50,429 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:51,163 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:51,429 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:52,459 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:53,312 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:38:53,313 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 with policy now |
| 2025-01-17 14:38:53,321 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:38:53,321 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:38:53,322 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:38:53,322 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:38:53,441 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:53,554 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/6myrxrf9-media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 |
| 2025-01-17 14:38:53,785 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:53,785 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:38:53,785 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 |
| 2025-01-17 14:38:53,785 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:38:54,080 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/k8nb6l9z-media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 |
| 2025-01-17 14:38:54,463 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:54,785 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:55,460 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:55,786 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:56,399 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:38:56,465 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:57,495 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:57,786 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:38:58,471 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:38:59,474 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:00,476 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:01,479 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:01,788 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:02,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:02,491 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:02,659 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:39:02,759 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:39:02,759 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:39:02,765 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:39:02,788 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:39:02,789 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:39:03,061 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:39:03,061 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:39:03,493 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:03,789 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:39:03,789 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:04,493 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:05,495 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:05,790 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:06,495 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:07,497 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:07,790 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:08,061 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:08,502 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:09,519 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:10,528 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:11,524 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:12,552 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:13,123 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:13,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:13,793 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:14,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:15,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:15,794 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:16,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:17,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:18,046 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:39:18,047 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:39:18,181 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:18,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:19,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:20,513 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:21,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:22,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:23,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:24,100 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:24,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:25,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:26,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:27,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:28,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:29,100 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:29,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:30,514 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:31,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:32,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:32,660 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:39:33,047 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:39:33,047 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:39:33,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:34,197 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:34,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:35,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:36,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:37,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:38,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:39,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:40,101 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:40,515 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:41,530 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:41,619 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:39:41,619 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 with policy now |
| 2025-01-17 14:39:41,634 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:39:41,635 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:39:41,635 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:39:41,635 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:39:41,804 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:41,804 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:39:41,804 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 |
| 2025-01-17 14:39:41,804 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:39:41,848 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/vmjffqzc-media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 |
| 2025-01-17 14:39:42,122 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/ajxs2v80-media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 |
| 2025-01-17 14:39:42,528 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:42,804 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:43,530 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:43,805 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:44,532 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:45,122 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:45,530 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:46,556 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:47,556 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:47,806 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:48,079 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:39:48,079 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:39:48,538 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:49,552 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:49,807 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:50,509 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:50,554 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:51,553 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:51,808 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:39:51,821 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:39:51,821 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:39:51,822 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:39:52,555 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:52,809 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:39:52,809 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:39:53,558 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:53,809 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:54,585 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:55,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:56,056 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:39:56,568 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:57,589 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:57,811 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:39:58,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:39:59,585 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:00,583 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:01,579 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:01,812 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:01,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:02,579 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:02,660 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:40:03,047 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:40:03,048 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:40:03,579 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:04,579 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:05,579 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:06,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:07,228 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:07,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:08,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:09,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:10,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:11,580 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:12,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:13,121 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:13,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:14,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:15,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:16,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:17,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:18,047 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:40:18,048 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:40:18,184 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:18,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:19,581 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:20,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:21,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:22,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:23,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:24,122 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:24,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:25,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:26,582 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:27,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:28,610 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:28,944 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:40:28,944 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 with policy now |
| 2025-01-17 14:40:28,961 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:40:28,961 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:40:28,961 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:40:28,962 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:40:29,190 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/e51sumpa-media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 |
| 2025-01-17 14:40:29,595 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:29,823 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:29,824 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:40:29,824 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 |
| 2025-01-17 14:40:29,824 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:40:30,019 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:30,123 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/ikvanlqv-media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 |
| 2025-01-17 14:40:30,601 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:30,824 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:31,615 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:32,615 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:32,661 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:40:33,077 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:40:33,077 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:40:33,636 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:33,825 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:34,616 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:35,203 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:35,648 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:36,649 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:37,649 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:37,827 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:38,635 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:38,827 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:40:38,880 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:40:38,881 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:40:38,881 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:40:39,088 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/df3m9gyi-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:40:39,647 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:39,828 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:39,828 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:40:40,284 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:40,643 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:41,638 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:42,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:43,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:43,829 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:40:44,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:45,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:46,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:46,621 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:47,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:48,050 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:40:48,050 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:40:48,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:49,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:50,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:51,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:52,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:52,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:53,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:54,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:55,622 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:56,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:57,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:40:57,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:58,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:40:59,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:00,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:01,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:02,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:02,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:02,661 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:41:03,050 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:41:03,050 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:41:03,623 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:04,624 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:05,624 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:06,624 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:07,624 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:08,152 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:08,624 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:09,645 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:10,628 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:10,929 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:41:10,929 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 with policy now |
| 2025-01-17 14:41:10,959 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:41:10,960 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:41:10,960 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:41:10,960 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:41:11,157 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/0tso4q2r-media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 |
| 2025-01-17 14:41:11,690 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:11,841 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:11,841 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:41:11,841 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 |
| 2025-01-17 14:41:11,841 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:41:12,146 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/z4k9ieig-media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 |
| 2025-01-17 14:41:12,681 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:12,841 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:13,177 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:13,649 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:14,640 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:15,665 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:15,843 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:16,662 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:17,657 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:18,060 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:41:18,060 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:41:18,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:18,682 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:19,668 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:19,844 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:20,686 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:20,845 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:41:20,933 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:41:20,934 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:41:20,934 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:41:21,685 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:21,845 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:21,845 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:41:22,696 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:23,699 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:23,846 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:24,178 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:24,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:25,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:25,847 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:26,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:27,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:28,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:29,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:30,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:30,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:31,674 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:32,661 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:41:32,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:33,050 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:41:33,051 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:41:33,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:34,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:35,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:35,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:36,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:37,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:38,675 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:39,676 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:40,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:40,676 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:41,676 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:42,676 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:43,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:44,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:45,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:46,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:46,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:47,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:48,051 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:41:48,051 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:41:48,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:49,677 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:50,678 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:51,174 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:51,678 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:52,691 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:52,819 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:41:52,819 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 with policy now |
| 2025-01-17 14:41:52,860 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 |
| 2025-01-17 14:41:52,861 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:41:52,861 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:41:52,861 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:41:52,861 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:41:52,862 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:41:53,046 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/w2acgh58-media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 |
| 2025-01-17 14:41:53,359 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/r6js5gee-media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 |
| 2025-01-17 14:41:53,694 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:53,861 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:53,861 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:41:54,687 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:55,719 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:56,187 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:41:56,693 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:57,714 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:57,862 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:41:58,714 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:41:59,717 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:00,713 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:01,723 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:01,864 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:01,901 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:02,662 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:42:02,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:03,082 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:42:03,083 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:42:03,753 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:04,748 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:05,769 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:06,748 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:06,866 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:42:06,990 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:42:06,990 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:42:06,990 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:06,990 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:42:07,772 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:07,866 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:07,867 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:42:08,771 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:09,777 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:09,867 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:10,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:11,776 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:11,868 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:12,786 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:12,920 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:13,775 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:14,802 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:15,789 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:16,776 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:17,787 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:18,071 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:42:18,071 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:42:18,162 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:18,789 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:19,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:19,872 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:20,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:21,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:21,873 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:22,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:23,187 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:23,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:24,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:25,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:26,778 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:27,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:28,187 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:28,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:29,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:30,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:31,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:32,662 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:42:32,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:33,054 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:42:33,054 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:42:33,208 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:33,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:33,878 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:34,779 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:35,780 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:36,780 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:37,780 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:38,780 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:39,187 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:39,782 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:40,781 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:41,781 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:42,781 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:43,782 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:44,188 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:44,782 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:45,782 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:46,808 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:47,786 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:47,884 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:47,973 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:42:47,973 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 with policy now |
| 2025-01-17 14:42:47,981 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:42:47,981 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:42:47,981 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:42:47,982 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:42:48,059 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:42:48,059 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:42:48,200 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/8tzr7loo-media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 |
| 2025-01-17 14:42:48,842 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:48,884 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:48,885 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:42:48,885 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 |
| 2025-01-17 14:42:48,885 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:42:49,199 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/8tgwv9hd-media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 |
| 2025-01-17 14:42:49,210 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:49,833 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:49,885 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:50,845 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:51,848 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:52,837 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:53,854 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:53,886 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:54,222 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:54,851 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:55,831 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:55,887 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:42:56,837 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:57,836 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:57,888 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:42:58,049 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:42:58,049 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:42:58,050 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:42:58,140 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/qc3rpfq7-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:42:58,831 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:58,889 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:42:59,878 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:42:59,881 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:42:59,889 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:00,842 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:01,848 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:02,663 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:43:02,865 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:03,077 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:43:03,077 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:43:03,851 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:03,891 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:04,861 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:05,201 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:05,841 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:06,860 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:07,878 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:08,902 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:09,880 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:10,211 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:10,875 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:11,866 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:12,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:13,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:13,895 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:14,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:15,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:16,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:16,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:17,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:18,056 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:43:18,056 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:43:18,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:19,867 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:20,868 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:21,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:21,868 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:22,868 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:23,868 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:24,868 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:25,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:26,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:27,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:27,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:28,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:29,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:30,869 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:31,870 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:32,200 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:32,663 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:43:32,870 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:33,056 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:43:33,057 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:43:33,870 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:34,870 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:35,871 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:36,871 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:37,201 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:37,871 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:38,871 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:39,906 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:40,644 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:43:40,644 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 with policy now |
| 2025-01-17 14:43:40,649 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:43:40,650 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:43:40,650 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:43:40,737 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:43:40,884 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/r1qx0bkk-media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 |
| 2025-01-17 14:43:40,901 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:40,907 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:40,907 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:43:40,907 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 |
| 2025-01-17 14:43:40,908 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:43:41,179 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/0qjlko87-media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 |
| 2025-01-17 14:43:41,900 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:41,908 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:42,810 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:42,908 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:42,933 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:43,895 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:43,908 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:44,910 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:45,913 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:46,910 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:46,937 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:47,919 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:48,056 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:43:48,057 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:43:48,146 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:48,936 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:49,958 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:50,911 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:50,993 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:51,111 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:43:51,111 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:43:51,111 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:43:51,912 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:43:51,912 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:43:51,959 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:52,912 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:52,989 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:53,226 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:43:53,970 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:54,913 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:55,019 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:55,957 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:56,960 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:57,962 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:58,915 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:43:59,000 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:43:59,211 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:00,017 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:01,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:01,994 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:02,664 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:44:03,021 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:03,078 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:44:03,079 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:44:03,991 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:04,229 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:05,009 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:05,985 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:06,918 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:06,985 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:07,985 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:08,985 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:09,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:10,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:10,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:11,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:12,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:13,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:14,986 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:15,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:15,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:16,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:17,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:18,058 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:44:18,058 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:44:18,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:19,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:20,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:20,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:21,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:22,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:23,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:24,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:25,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:26,206 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:26,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:27,987 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:28,988 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:29,988 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:30,988 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:31,207 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:31,988 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:32,664 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:44:32,996 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:33,131 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:44:33,131 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:44:33,696 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:44:33,696 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 with policy now |
| 2025-01-17 14:44:33,711 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:44:33,711 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:44:33,712 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:44:33,719 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:44:33,915 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/a81qmh8j-media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 |
| 2025-01-17 14:44:33,930 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:44:33,930 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 |
| 2025-01-17 14:44:33,931 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:44:33,992 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:34,207 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/43vz04pg-media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 |
| 2025-01-17 14:44:34,931 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:35,003 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:35,931 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:36,012 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:36,834 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:36,932 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:37,020 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:38,008 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:38,932 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:39,053 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:40,036 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:40,933 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:41,015 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:41,905 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:42,022 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:42,934 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:43,049 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:43,935 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:44:44,046 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:44,176 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:44:44,177 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:44:44,177 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:44:44,935 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:44,935 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:44:45,058 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:46,043 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:46,936 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:46,993 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:47,049 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:48,061 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:48,094 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:44:48,095 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:44:49,054 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:50,050 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:50,938 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:44:51,046 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:52,051 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:52,490 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:53,053 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:54,108 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:55,136 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:56,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:57,107 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:58,037 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:44:58,104 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:44:59,096 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:00,096 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:00,942 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:01,096 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:02,096 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:02,665 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:45:02,943 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:03,070 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:45:03,070 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:45:03,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:03,218 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:04,096 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:04,944 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:05,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:06,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:07,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:08,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:08,236 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:09,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:10,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:11,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:12,097 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:13,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:13,237 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:14,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:15,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:16,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:16,949 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:17,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:18,071 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:45:18,071 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:45:18,113 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:19,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:19,236 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:20,098 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:21,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:22,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:23,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:24,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:25,099 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:25,237 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:26,119 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:26,922 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:45:26,922 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 with policy now |
| 2025-01-17 14:45:26,922 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:45:26,923 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:45:26,923 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:26,925 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:26,954 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:26,955 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:26,955 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 |
| 2025-01-17 14:45:26,955 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval |
| 2025-01-17 14:45:27,105 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:27,155 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/0ehdgalh-media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 |
| 2025-01-17 14:45:27,434 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/mlf2r53b-media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 |
| 2025-01-17 14:45:28,157 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:28,955 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:29,139 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:29,956 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:30,185 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:30,260 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:31,130 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:32,165 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:32,665 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:45:32,957 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:33,089 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-17 14:45:33,089 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-17 14:45:33,176 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:34,179 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:35,145 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:35,493 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:35,958 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:45:36,179 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:36,235 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:45:36,235 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:36,236 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:36,951 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 14:45:36,951 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 14:45:36,959 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:36,959 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:36,959 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json |
| 2025-01-17 14:45:36,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media |
| 2025-01-17 14:45:36,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval |
| 2025-01-17 14:45:36,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table |
| 2025-01-17 14:45:37,110 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:45:37,147 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:37,148 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/myl9s5ei-media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json |
| 2025-01-17 14:45:37,428 INFO wandb-upload_2:2514330 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpph57zsi3 |
| 2025-01-17 14:45:37,482 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-17 14:45:37,497 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-17 14:45:37,810 INFO SenderThread:2514330 [sender.py:send_request_log_artifact():1525] logged artifact run-odxnffav-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTEwMzMyNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTU0NTk4OQ==', 'latestArtifact': None}} |
| 2025-01-17 14:45:37,810 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:45:37,810 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json with policy now |
| 2025-01-17 14:45:37,810 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:45:37,810 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,811 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,811 DEBUG SenderThread:2514330 [sender.py:send():379] send: telemetry |
| 2025-01-17 14:45:37,811 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,815 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,815 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,816 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,816 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,816 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,816 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,817 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,817 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,817 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,817 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,818 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,818 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,818 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,818 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,819 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,819 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,819 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,820 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,820 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,820 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:37,820 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:37,821 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-17 14:45:37,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:45:37,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:37,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json |
| 2025-01-17 14:45:37,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval |
| 2025-01-17 14:45:38,055 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/2nmrwxif-media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json |
| 2025-01-17 14:45:38,148 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:38,189 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/dv04cvip-media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json |
| 2025-01-17 14:45:38,223 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/p2c0lp91-events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:45:38,234 INFO wandb-upload_2:2514330 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpl5qywjbs |
| 2025-01-17 14:45:38,649 INFO SenderThread:2514330 [sender.py:send_request_log_artifact():1525] logged artifact run-odxnffav-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ0OTEwMzM3Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MTU0NjAzMA==', 'latestArtifact': None}} |
| 2025-01-17 14:45:38,649 DEBUG SenderThread:2514330 [sender.py:send():379] send: files |
| 2025-01-17 14:45:38,649 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json with policy now |
| 2025-01-17 14:45:38,649 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:45:38,650 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:38,650 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:38,904 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /tmp/tmpawxzwn98wandb/aw88p4pp-media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json |
| 2025-01-17 14:45:38,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:38,960 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:39,148 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:40,148 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:40,651 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:40,961 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:41,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:42,149 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:43,025 DEBUG SenderThread:2514330 [sender.py:send():379] send: exit |
| 2025-01-17 14:45:43,025 INFO SenderThread:2514330 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-17 14:45:43,026 INFO SenderThread:2514330 [sender.py:send_exit():588] handling runtime: 970 |
| 2025-01-17 14:45:43,026 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:43,026 INFO SenderThread:2514330 [sender.py:send_exit():594] send defer |
| 2025-01-17 14:45:43,026 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:43,026 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-17 14:45:43,026 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:43,026 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-17 14:45:43,026 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-17 14:45:43,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:43,027 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-17 14:45:43,027 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:43,027 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-17 14:45:43,027 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-17 14:45:43,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:43,027 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-17 14:45:43,027 INFO HandlerThread:2514330 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-17 14:45:43,027 INFO HandlerThread:2514330 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-17 14:45:43,027 INFO HandlerThread:2514330 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-17 14:45:43,028 DEBUG SystemMonitor:2514330 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-17 14:45:43,028 DEBUG SystemMonitor:2514330 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-17 14:45:43,079 INFO HandlerThread:2514330 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-17 14:45:43,079 INFO HandlerThread:2514330 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-17 14:45:43,079 INFO HandlerThread:2514330 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-17 14:45:43,080 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:43,080 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-17 14:45:43,080 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-17 14:45:43,080 DEBUG SenderThread:2514330 [sender.py:send():379] send: stats |
| 2025-01-17 14:45:43,080 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:43,080 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-17 14:45:43,080 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:43,081 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-17 14:45:43,081 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-17 14:45:43,081 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:43,081 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-17 14:45:43,963 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:44,964 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:46,965 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/config.yaml |
| 2025-01-17 14:45:49,242 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:49,243 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:49,243 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-17 14:45:49,243 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:49,244 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-17 14:45:49,244 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-17 14:45:49,244 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:49,244 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:49,244 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-17 14:45:49,244 DEBUG SenderThread:2514330 [sender.py:send():379] send: history |
| 2025-01-17 14:45:49,245 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-17 14:45:49,246 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:49,246 DEBUG SenderThread:2514330 [sender.py:send():379] send: summary |
| 2025-01-17 14:45:49,249 INFO SenderThread:2514330 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-17 14:45:49,250 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:49,250 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-17 14:45:49,250 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-17 14:45:49,250 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:49,250 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-17 14:45:49,250 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:49,250 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-17 14:45:49,250 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-17 14:45:49,250 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-17 14:45:49,250 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:49,250 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-17 14:45:49,251 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:49,251 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-17 14:45:49,966 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:50,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:51,043 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-17 14:45:51,044 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:51,044 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:51,044 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-17 14:45:51,044 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:51,044 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-17 14:45:51,044 INFO SenderThread:2514330 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-17 14:45:51,047 INFO SenderThread:2514330 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-17 14:45:51,049 INFO SenderThread:2514330 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-17 14:45:51,068 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-17 14:45:51,068 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:51,068 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-17 14:45:51,068 DEBUG SenderThread:2514330 [sender.py:send():379] send: artifact |
| 2025-01-17 14:45:51,751 INFO wandb-upload_2:2514330 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp5pworv_0 |
| 2025-01-17 14:45:51,752 INFO wandb-upload_3:2514330 [upload_job.py:push():85] Skipped uploading /tmp/tmpflclq4of/wandb-job.json |
| 2025-01-17 14:45:51,872 INFO wandb-upload_0:2514330 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpxz7r8926 |
| 2025-01-17 14:45:51,967 INFO Thread-12 :2514330 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:52,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:52,466 INFO SenderThread:2514330 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ0OTEwNDY1OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ0NDE0MDQxMg==', 'versionIndex': 441}}} |
| 2025-01-17 14:45:52,466 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:52,466 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-17 14:45:52,467 INFO SenderThread:2514330 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-17 14:45:52,968 INFO SenderThread:2514330 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files |
| 2025-01-17 14:45:52,968 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/config.yaml config.yaml |
| 2025-01-17 14:45:52,969 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json wandb-summary.json |
| 2025-01-17 14:45:52,970 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/requirements.txt requirements.txt |
| 2025-01-17 14:45:52,970 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-17 14:45:52,970 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 events.out.tfevents.1737142173.leonmkim-ROG-Strix-G15CS-G15CS.2514148.0 |
| 2025-01-17 14:45:52,970 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-17 14:45:52,971 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log output.log |
| 2025-01-17 14:45:52,971 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/diff.patch diff.patch |
| 2025-01-17 14:45:52,971 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 media/videos/eval/17_eval_17_ba7e994f26a18d93d39e.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 media/videos/eval/13_eval_13_4585080ffb0d13dad794.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 media/videos/eval/8_eval_8_c97055dba6783f8bd296.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 media/videos/eval/15_eval_15_8bb20ba2a7c41db7ced9.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 media/videos/eval/9_eval_9_1ad291b6cce26da44467.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 media/videos/eval/7_eval_7_b6f988d87c4abf5cf1bd.mp4 |
| 2025-01-17 14:45:52,975 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 media/videos/eval/5_eval_5_b5cc7767f4cc99909240.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 media/videos/eval/10_eval_10_75a622b28aa17519a714.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 media/videos/eval/14_eval_14_9c74466fc3e808dda0ce.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 media/videos/eval/11_eval_11_e5a9c02b219899b0e912.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 media/videos/eval/0_eval_0_9913e877ba79f187dfce.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 media/videos/eval/2_eval_2_74f6d531a1bae5354927.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 media/videos/eval/12_eval_12_6c8ef51bd0e32f8f77f2.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 media/videos/eval/6_eval_6_507e5204685e3d527cd3.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 media/videos/eval/3_eval_3_90efa260cbf26eb4711e.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 media/videos/eval/1_eval_1_4d7ecef067735039e088.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 media/videos/eval/18_eval_18_d49a7e0e6acc433fa848.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 media/videos/eval/19_eval_19_b74266e52a4c9294c18d.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 media/videos/eval/4_eval_4_87ec98a2ed597fa45d1d.mp4 |
| 2025-01-17 14:45:52,976 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 media/videos/eval/16_eval_16_2f6b9b3e7e132b67ed2e.mp4 |
| 2025-01-17 14:45:52,977 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json media/table/eval/success_rate_ci_20_52d7fd349d2c5dd77c8a.table.json |
| 2025-01-17 14:45:52,977 INFO SenderThread:2514330 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json media/table/eval/total_success_rate_ci_21_4a20749ae6ca4d3c4e22.table.json |
| 2025-01-17 14:45:52,977 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-17 14:45:52,977 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:52,977 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:52,977 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-17 14:45:52,977 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:52,977 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-17 14:45:52,977 INFO SenderThread:2514330 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 14:45:53,027 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:53,028 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:53,159 INFO wandb-upload_0:2514330 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/requirements.txt |
| 2025-01-17 14:45:53,161 INFO wandb-upload_3:2514330 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/wandb-summary.json |
| 2025-01-17 14:45:53,161 INFO wandb-upload_1:2514330 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/config.yaml |
| 2025-01-17 14:45:53,228 INFO wandb-upload_2:2514330 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/files/output.log |
| 2025-01-17 14:45:53,428 INFO Thread-11 (_thread_body):2514330 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-17 14:45:53,428 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:53,429 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-17 14:45:53,429 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:53,429 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-17 14:45:53,429 INFO SenderThread:2514330 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-17 14:45:53,434 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-17 14:45:53,434 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:53,434 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-17 14:45:53,434 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:53,434 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-17 14:45:53,435 INFO SenderThread:2514330 [file_stream.py:finish():601] file stream finish called |
| 2025-01-17 14:45:53,699 INFO SenderThread:2514330 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-17 14:45:53,699 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-17 14:45:53,699 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:53,699 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-17 14:45:53,699 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:53,699 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-17 14:45:53,700 INFO SenderThread:2514330 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-17 14:45:53,700 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-17 14:45:53,700 INFO HandlerThread:2514330 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-17 14:45:53,700 DEBUG SenderThread:2514330 [sender.py:send():379] send: final |
| 2025-01-17 14:45:53,700 DEBUG SenderThread:2514330 [sender.py:send():379] send: footer |
| 2025-01-17 14:45:53,700 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: defer |
| 2025-01-17 14:45:53,700 INFO SenderThread:2514330 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-17 14:45:53,701 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:53,701 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:53,702 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-17 14:45:53,702 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-17 14:45:53,702 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: server_info |
| 2025-01-17 14:45:53,702 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-17 14:45:53,704 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-17 14:45:53,705 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-17 14:45:53,766 DEBUG SenderThread:2514330 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-17 14:45:53,766 INFO MainThread:2514330 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-17 14:45:53,767 INFO MainThread:2514330 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-17 14:45:53,767 INFO MainThread:2514330 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-17 14:45:53,767 DEBUG HandlerThread:2514330 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-17 14:45:53,767 INFO HandlerThread:2514330 [handler.py:finish():882] shutting down handler |
| 2025-01-17 14:45:54,702 INFO WriterThread:2514330 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3456_0/odxnffav/wandb/run-20250117_142932-odxnffav/run-odxnffav.wandb |
| 2025-01-17 14:45:54,766 INFO SenderThread:2514330 [sender.py:finish():1615] shutting down sender |
| 2025-01-17 14:45:54,766 INFO SenderThread:2514330 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-17 14:45:54,766 INFO SenderThread:2514330 [file_pusher.py:join():175] waiting for file pusher |
|
|
