| 2025-01-06 17:28:47,455 INFO StreamThr :2090663 [internal.py:wandb_internal():85] W&B internal server running at pid: 2090663, started at: 2025-01-06 17:28:47.455096 |
| 2025-01-06 17:28:47,456 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 17:28:47,468 INFO WriterThread:2090663 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/run-d4l0r92a.wandb |
| 2025-01-06 17:28:47,468 DEBUG SenderThread:2090663 [sender.py:send():379] send: header |
| 2025-01-06 17:28:47,477 DEBUG SenderThread:2090663 [sender.py:send():379] send: run |
| 2025-01-06 17:28:47,747 INFO SenderThread:2090663 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files |
| 2025-01-06 17:28:47,747 INFO SenderThread:2090663 [sender.py:_start_run_threads():1188] run started: d4l0r92a with start time 1736202527.45553 |
| 2025-01-06 17:28:47,763 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 17:28:47,763 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 17:28:47,839 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 17:28:47,843 DEBUG HandlerThread:2090663 [system_info.py:__init__():26] System info init |
| 2025-01-06 17:28:47,844 DEBUG HandlerThread:2090663 [system_info.py:__init__():41] System info init done |
| 2025-01-06 17:28:47,844 INFO HandlerThread:2090663 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 17:28:47,844 INFO SystemMonitor:2090663 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 17:28:47,844 INFO HandlerThread:2090663 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 17:28:47,844 INFO SystemMonitor:2090663 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 17:28:47,844 INFO SystemMonitor:2090663 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 17:28:47,845 INFO SystemMonitor:2090663 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 17:28:47,845 INFO SystemMonitor:2090663 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 17:28:47,846 INFO SystemMonitor:2090663 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 17:28:47,856 DEBUG HandlerThread:2090663 [system_info.py:probe():152] Probing system |
| 2025-01-06 17:28:47,858 DEBUG HandlerThread:2090663 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 17:28:47,863 DEBUG HandlerThread:2090663 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 17:28:47,863 DEBUG HandlerThread:2090663 [system_info.py:probe():200] Probing system done |
| 2025-01-06 17:28:47,863 DEBUG HandlerThread:2090663 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-06T22:28:47.856980', 'startedAt': '2025-01-06T22:28:47.447626', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.221375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4497.996, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 340.60540771484375}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 17:28:47,863 INFO HandlerThread:2090663 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 17:28:47,863 INFO HandlerThread:2090663 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 17:28:47,863 DEBUG HandlerThread:2090663 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 17:28:47,864 DEBUG HandlerThread:2090663 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 17:28:47,864 DEBUG HandlerThread:2090663 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 17:28:47,929 DEBUG HandlerThread:2090663 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 17:28:47,929 INFO HandlerThread:2090663 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 17:28:47,934 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:28:47,935 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 17:28:47,935 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 17:28:47,935 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 17:28:48,174 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/h3n6188w-wandb-metadata.json |
| 2025-01-06 17:28:48,187 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 17:28:48,187 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 17:28:48,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:28:48,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:28:48,206 INFO wandb-upload_1:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/h5szt28m-code/FISH/eval_robot.py |
| 2025-01-06 17:28:48,207 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/lqn5ijm1-diff.patch |
| 2025-01-06 17:28:48,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:48,233 INFO HandlerThread:2090663 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/tb" |
| save: true |
| } |
| control { |
| relay_id: "53cad69e1a284cc0844ac292d4426ea7" |
| } |
| _info { |
| stream_id: "d4l0r92a" |
| } |
|
|
| 2025-01-06 17:28:48,233 DEBUG HandlerThread:2090663 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 17:28:48,263 DEBUG SenderThread:2090663 [sender.py:send():379] send: telemetry |
| 2025-01-06 17:28:48,284 DEBUG SenderThread:2090663 [sender.py:send():379] send: tbrecord |
| 2025-01-06 17:28:48,285 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:28:48,285 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 with policy live |
| 2025-01-06 17:28:48,471 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/dbm412tw-events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:28:48,749 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/requirements.txt |
| 2025-01-06 17:28:48,749 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-metadata.json |
| 2025-01-06 17:28:48,749 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:28:48,749 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:28:48,750 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/diff.patch |
| 2025-01-06 17:28:48,750 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/code/FISH/eval_robot.py |
| 2025-01-06 17:28:48,750 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/code |
| 2025-01-06 17:28:48,750 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/code/FISH |
| 2025-01-06 17:28:49,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:50,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:50,748 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:28:51,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:52,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:53,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:28:53,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:54,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:55,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:56,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:56,787 DEBUG SenderThread:2090663 [sender.py:send():379] send: config |
| 2025-01-06 17:28:57,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:58,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:28:58,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:28:58,751 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:28:59,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:00,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:00,751 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:01,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:02,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:02,752 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:03,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:29:03,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:29:03,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:03,324 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:04,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:05,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:06,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:06,753 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:07,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:08,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:09,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:09,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:10,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:10,754 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:11,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:12,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:12,755 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:13,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:14,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:14,393 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:14,755 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:15,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:16,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:17,192 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:18,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:29:18,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:29:18,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:19,192 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:19,861 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:20,192 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:20,757 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/config.yaml |
| 2025-01-06 17:29:20,757 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:21,192 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:22,205 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:23,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:24,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:24,758 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:25,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:25,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:26,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:27,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:28,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:29,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:30,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:31,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:31,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:32,194 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:33,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:29:33,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:29:33,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:34,194 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:35,194 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:36,194 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:36,265 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:37,194 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:37,445 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:29:37,445 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 with policy now |
| 2025-01-06 17:29:37,445 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:29:37,446 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:29:37,447 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:29:37,447 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:29:37,702 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/5j3gsj46-media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 |
| 2025-01-06 17:29:37,761 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 |
| 2025-01-06 17:29:37,762 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:29:37,762 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos |
| 2025-01-06 17:29:37,762 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media |
| 2025-01-06 17:29:37,762 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval |
| 2025-01-06 17:29:38,016 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/tstaoxft-media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 |
| 2025-01-06 17:29:38,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:38,762 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:39,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:40,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:41,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:41,597 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:42,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:42,763 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:43,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:44,195 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:44,764 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:29:45,196 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:46,196 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:47,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:47,306 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:47,846 DEBUG SystemMonitor:2090663 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 17:29:47,847 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:29:48,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:29:48,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:29:48,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:49,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:50,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:51,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:52,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:52,309 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:53,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:54,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:55,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:56,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:57,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:58,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:29:58,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:29:59,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:00,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:01,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:02,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:02,770 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:30:03,012 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/gcd6wjds-events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:30:03,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:30:03,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:30:03,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:03,261 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:04,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:04,770 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:05,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:06,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:07,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:08,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:08,772 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:09,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:09,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:10,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:11,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:12,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:13,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:14,200 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:15,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:15,201 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:16,201 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:17,201 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:17,847 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:30:18,188 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:30:18,188 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:30:18,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:19,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:20,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:20,773 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:20,775 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:21,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:22,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:23,203 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:24,203 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:24,777 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:25,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:26,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:26,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:27,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:28,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:29,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:30,090 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:30:30,090 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 with policy now |
| 2025-01-06 17:30:30,091 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:30:30,091 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:30:30,091 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:30:30,093 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:30:30,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:30,335 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/6qglpwso-media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 |
| 2025-01-06 17:30:30,779 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:30:30,779 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:30,779 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 |
| 2025-01-06 17:30:30,779 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval |
| 2025-01-06 17:30:31,058 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/i569hmqq-media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 |
| 2025-01-06 17:30:31,208 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:32,101 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:32,206 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:32,779 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:33,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:30:33,189 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:30:33,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:34,216 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:34,780 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:35,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:36,209 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:37,210 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:38,065 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:38,210 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:38,781 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:39,210 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:40,210 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:41,212 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:42,211 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:43,192 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:43,211 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:44,210 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:45,212 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:46,211 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:47,211 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:47,847 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:30:48,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:30:48,191 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:30:48,237 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:48,316 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:49,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:50,212 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:51,212 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:52,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:53,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:53,602 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:53,786 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:30:54,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:54,736 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:30:54,736 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:30:54,737 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:30:54,786 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:30:54,787 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:55,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:56,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:56,787 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:57,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:58,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:30:58,788 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:30:59,193 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:30:59,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:00,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:01,213 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:02,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:03,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:31:03,189 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:31:03,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:04,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:04,319 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:05,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:06,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:07,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:08,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:09,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:10,204 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:10,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:11,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:12,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:13,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:14,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:15,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:16,199 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:16,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:17,215 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:17,848 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:31:18,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:31:18,189 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:31:18,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:19,216 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:20,216 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:21,217 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:21,281 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:22,216 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:23,217 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:24,217 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:25,219 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:25,829 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:31:25,829 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 with policy now |
| 2025-01-06 17:31:25,829 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:31:25,830 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:31:25,830 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:31:25,831 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:31:26,091 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/fwfcg65b-media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 |
| 2025-01-06 17:31:26,223 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:26,797 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:31:26,797 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:26,798 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 |
| 2025-01-06 17:31:26,798 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval |
| 2025-01-06 17:31:26,851 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:27,096 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/gmh3byrr-media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 |
| 2025-01-06 17:31:27,219 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:28,226 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:29,221 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:30,226 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:30,799 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:31,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:31,865 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:32,224 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:32,799 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:33,198 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:31:33,198 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:31:33,245 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:34,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:35,227 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:36,230 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:37,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:37,324 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:37,802 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:31:38,047 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/heuqkeu5-events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:31:38,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:38,493 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:31:38,493 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:31:38,494 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:31:38,802 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:31:38,803 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:39,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:40,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:40,804 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:41,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:42,229 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:42,805 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:42,971 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:43,230 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:44,230 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:45,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:46,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:47,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:47,848 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:31:48,108 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:48,189 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:31:48,189 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:31:48,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:49,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:50,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:50,808 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:51,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:52,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:53,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:53,231 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:54,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:55,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:56,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:56,810 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:31:57,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:58,202 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:31:58,232 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:31:59,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:00,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:01,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:02,233 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:03,196 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:32:03,196 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:32:03,235 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:03,306 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:03,760 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:32:03,760 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 with policy now |
| 2025-01-06 17:32:03,761 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:32:03,763 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:32:03,763 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:32:03,764 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:32:03,814 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:32:03,814 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 |
| 2025-01-06 17:32:03,814 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval |
| 2025-01-06 17:32:04,011 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/1buyzstt-media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 |
| 2025-01-06 17:32:04,234 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:04,262 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/a4v633zo-media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 |
| 2025-01-06 17:32:04,814 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:05,247 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:06,237 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:07,240 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:08,242 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:08,816 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:09,214 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:09,244 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:10,241 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:10,816 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:11,245 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:12,241 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:13,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:14,245 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:15,208 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:15,246 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:16,247 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:17,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:17,849 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:32:18,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:32:18,197 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:32:18,253 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:19,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:20,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:20,263 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:21,257 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:22,253 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:23,250 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:23,547 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:32:23,547 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:32:23,547 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:32:23,820 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:32:23,821 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:32:24,255 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:24,821 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:25,258 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:25,501 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:26,256 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:26,822 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:27,289 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:28,250 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:28,822 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:29,250 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:30,250 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:31,207 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:31,250 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:32,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:33,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:32:33,190 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:32:33,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:34,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:34,824 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:35,251 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:36,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:36,853 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:37,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:38,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:39,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:40,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:41,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:42,207 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:42,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:43,252 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:44,253 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:45,253 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:46,253 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:47,208 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:47,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:47,850 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:32:48,190 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:32:48,190 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:32:48,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:49,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:50,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:51,254 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:52,255 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:52,342 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:53,255 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:54,255 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:55,273 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:55,467 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:32:55,467 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 with policy now |
| 2025-01-06 17:32:55,470 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:32:55,470 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:32:55,470 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:32:55,471 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:32:55,723 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/k6jof3hy-media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 |
| 2025-01-06 17:32:55,831 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:32:55,832 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 |
| 2025-01-06 17:32:55,832 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval |
| 2025-01-06 17:32:56,096 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/2dsre9dk-media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 |
| 2025-01-06 17:32:56,261 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:56,832 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:32:57,264 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:57,501 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:32:58,268 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:32:59,271 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:00,271 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:00,833 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:01,287 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:02,299 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:02,834 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:03,197 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 17:33:03,198 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 17:33:03,266 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:33:03,276 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:03,834 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:33:04,076 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/jzc7jh33-events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:33:04,272 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:04,599 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:33:04,599 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:04,599 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:04,654 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 17:33:04,655 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval |
| 2025-01-06 17:33:04,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table |
| 2025-01-06 17:33:04,837 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:33:05,023 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/ev91wtea-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2025-01-06 17:33:05,120 INFO wandb-upload_2:2090663 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmplp0yrt7a |
| 2025-01-06 17:33:05,181 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 17:33:05,191 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 17:33:05,272 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:05,594 INFO SenderThread:2090663 [sender.py:send_request_log_artifact():1525] logged artifact run-d4l0r92a-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkzNjExMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxMjQ2OQ==', 'latestArtifact': None}} |
| 2025-01-06 17:33:05,594 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:33:05,594 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now |
| 2025-01-06 17:33:05,595 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:33:05,595 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,596 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,597 DEBUG SenderThread:2090663 [sender.py:send():379] send: telemetry |
| 2025-01-06 17:33:05,597 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,597 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,597 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,597 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,597 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,598 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,598 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,598 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,598 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,601 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,603 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,604 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,604 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,604 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,604 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,606 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,606 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,606 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,607 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,607 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,607 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:05,607 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:05,607 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 17:33:05,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:33:05,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:05,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json |
| 2025-01-06 17:33:05,836 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval |
| 2025-01-06 17:33:05,851 INFO wandb-upload_1:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/jwr53dwx-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2025-01-06 17:33:06,005 INFO wandb-upload_2:2090663 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpylcyyb7s |
| 2025-01-06 17:33:06,033 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/rz253b0p-media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json |
| 2025-01-06 17:33:06,272 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:06,389 INFO SenderThread:2090663 [sender.py:send_request_log_artifact():1525] logged artifact run-d4l0r92a-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMTkzNjE0OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDUxMjQ3Mg==', 'latestArtifact': None}} |
| 2025-01-06 17:33:06,389 DEBUG SenderThread:2090663 [sender.py:send():379] send: files |
| 2025-01-06 17:33:06,390 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json with policy now |
| 2025-01-06 17:33:06,390 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:33:06,390 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:06,391 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:06,638 INFO wandb-upload_1:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/x296pkve-media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json |
| 2025-01-06 17:33:06,835 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:06,836 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:07,272 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:07,994 DEBUG SenderThread:2090663 [sender.py:send():379] send: exit |
| 2025-01-06 17:33:07,994 INFO SenderThread:2090663 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 17:33:07,994 INFO SenderThread:2090663 [sender.py:send_exit():588] handling runtime: 260 |
| 2025-01-06 17:33:07,994 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:07,994 INFO SenderThread:2090663 [sender.py:send_exit():594] send defer |
| 2025-01-06 17:33:07,994 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:07,994 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 17:33:07,995 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:07,995 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 17:33:07,995 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 17:33:07,995 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:07,995 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 17:33:07,995 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:07,995 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 17:33:07,995 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 17:33:07,995 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:07,995 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 17:33:07,995 INFO HandlerThread:2090663 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 17:33:07,995 DEBUG SystemMonitor:2090663 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 17:33:07,995 DEBUG SystemMonitor:2090663 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 17:33:07,997 INFO HandlerThread:2090663 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 17:33:07,997 INFO HandlerThread:2090663 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 17:33:08,050 INFO HandlerThread:2090663 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 17:33:08,050 INFO HandlerThread:2090663 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 17:33:08,051 INFO HandlerThread:2090663 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 17:33:08,051 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:08,051 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 17:33:08,051 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 17:33:08,051 DEBUG SenderThread:2090663 [sender.py:send():379] send: stats |
| 2025-01-06 17:33:08,051 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:08,051 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 17:33:08,052 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:08,052 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 17:33:08,052 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 17:33:08,052 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:08,052 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 17:33:08,836 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:08,837 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:09,837 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/config.yaml |
| 2025-01-06 17:33:14,604 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:14,605 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:33:14,605 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 17:33:14,605 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:33:14,606 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:14,606 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 17:33:14,606 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 17:33:14,606 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:14,606 DEBUG SenderThread:2090663 [sender.py:send():379] send: history |
| 2025-01-06 17:33:14,606 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 17:33:14,606 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:14,607 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 17:33:14,607 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:14,607 DEBUG SenderThread:2090663 [sender.py:send():379] send: summary |
| 2025-01-06 17:33:14,611 INFO SenderThread:2090663 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 17:33:14,611 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:14,611 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 17:33:14,611 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 17:33:14,611 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:14,611 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 17:33:14,612 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:14,612 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 17:33:14,612 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 17:33:14,612 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:33:14,612 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:14,612 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 17:33:14,612 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:14,612 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 17:33:14,839 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:14,995 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:17,998 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 17:33:17,998 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:17,998 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:17,998 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 17:33:17,999 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:17,999 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 17:33:17,999 INFO SenderThread:2090663 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 17:33:18,001 INFO SenderThread:2090663 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 17:33:18,003 INFO SenderThread:2090663 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 17:33:18,013 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 17:33:18,013 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:18,013 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 17:33:18,014 DEBUG SenderThread:2090663 [sender.py:send():379] send: artifact |
| 2025-01-06 17:33:18,387 INFO wandb-upload_2:2090663 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp30y1vr2u |
| 2025-01-06 17:33:18,388 INFO wandb-upload_1:2090663 [upload_job.py:push():85] Skipped uploading /tmp/tmpelzxd0ux/wandb-job.json |
| 2025-01-06 17:33:18,481 INFO wandb-upload_0:2090663 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpf94pqpss |
| 2025-01-06 17:33:18,840 INFO Thread-12 :2090663 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:18,943 INFO SenderThread:2090663 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkzNjUwMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTY1Ng==', 'versionIndex': 423}}} |
| 2025-01-06 17:33:18,943 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:18,943 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 17:33:18,943 INFO SenderThread:2090663 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 17:33:18,996 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:19,841 INFO SenderThread:2090663 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files |
| 2025-01-06 17:33:19,841 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/config.yaml config.yaml |
| 2025-01-06 17:33:19,842 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 17:33:19,842 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/requirements.txt requirements.txt |
| 2025-01-06 17:33:19,842 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 17:33:19,842 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:33:19,843 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log output.log |
| 2025-01-06 17:33:19,844 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/diff.patch diff.patch |
| 2025-01-06 17:33:19,849 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 media/videos/eval/3_eval_3_cac182b4027a7521827c.mp4 |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 media/videos/eval/4_eval_4_12a3aaeba0fef5ed138d.mp4 |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 media/videos/eval/0_eval_0_80d7712c707615309a21.mp4 |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 media/videos/eval/2_eval_2_c6de2b4ebb693674e28c.mp4 |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 media/videos/eval/1_eval_1_92182b2f19c7840dbbc4.mp4 |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json media/table/eval/total_success_rate_ci_6_3fbfe773b6a6adb9e9f8.table.json |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2025-01-06 17:33:19,850 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 17:33:19,850 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:19,851 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:19,851 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 17:33:19,851 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 17:33:19,853 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:19,853 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 17:33:19,854 INFO SenderThread:2090663 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 17:33:19,996 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:19,997 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:20,033 INFO wandb-upload_2:2090663 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/config.yaml |
| 2025-01-06 17:33:20,036 INFO wandb-upload_0:2090663 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/wandb-summary.json |
| 2025-01-06 17:33:20,088 INFO wandb-upload_1:2090663 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/requirements.txt |
| 2025-01-06 17:33:20,109 INFO wandb-upload_3:2090663 [upload_job.py:push():130] Uploaded file /tmp/tmp6hnk1gbfwandb/sot7vj7s-events.out.tfevents.1736202528.leonmkim-ROG-Strix-G15CS-G15CS.2090492.0 |
| 2025-01-06 17:33:20,117 INFO wandb-upload_4:2090663 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/files/output.log |
| 2025-01-06 17:33:20,318 INFO Thread-11 (_thread_body):2090663 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 17:33:20,318 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:20,318 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 17:33:20,318 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:20,318 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 17:33:20,318 INFO SenderThread:2090663 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 17:33:20,319 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 17:33:20,319 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:20,320 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 17:33:20,320 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:20,320 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 17:33:20,320 INFO SenderThread:2090663 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 17:33:20,390 INFO SenderThread:2090663 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 17:33:20,390 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 17:33:20,390 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:20,391 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 17:33:20,391 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:20,391 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 17:33:20,391 INFO SenderThread:2090663 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 17:33:20,391 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 17:33:20,391 INFO HandlerThread:2090663 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 17:33:20,391 DEBUG SenderThread:2090663 [sender.py:send():379] send: final |
| 2025-01-06 17:33:20,391 DEBUG SenderThread:2090663 [sender.py:send():379] send: footer |
| 2025-01-06 17:33:20,391 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 17:33:20,391 INFO SenderThread:2090663 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 17:33:20,392 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:20,392 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 17:33:20,392 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 17:33:20,392 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 17:33:20,393 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:20,393 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 17:33:20,393 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 17:33:20,395 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 17:33:20,453 DEBUG SenderThread:2090663 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 17:33:20,454 INFO MainThread:2090663 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 17:33:20,454 INFO MainThread:2090663 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 17:33:20,455 INFO MainThread:2090663 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 17:33:20,455 DEBUG HandlerThread:2090663 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 17:33:20,455 INFO HandlerThread:2090663 [handler.py:finish():882] shutting down handler |
| 2025-01-06 17:33:21,393 INFO WriterThread:2090663 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/d4l0r92a/wandb/run-20250106_172847-d4l0r92a/run-d4l0r92a.wandb |
| 2025-01-06 17:33:21,453 INFO SenderThread:2090663 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 17:33:21,454 INFO SenderThread:2090663 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 17:33:21,454 INFO SenderThread:2090663 [file_pusher.py:join():175] waiting for file pusher |
|
|
