| 2025-01-03 22:45:23,762 INFO StreamThr :1987335 [internal.py:wandb_internal():85] W&B internal server running at pid: 1987335, started at: 2025-01-03 22:45:23.762311 |
| 2025-01-03 22:45:23,763 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status |
| 2025-01-03 22:45:23,774 INFO WriterThread:1987335 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/run-52k33ku2.wandb |
| 2025-01-03 22:45:23,775 DEBUG SenderThread:1987335 [sender.py:send():379] send: header |
| 2025-01-03 22:45:23,785 DEBUG SenderThread:1987335 [sender.py:send():379] send: run |
| 2025-01-03 22:45:24,051 INFO SenderThread:1987335 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files |
| 2025-01-03 22:45:24,051 INFO SenderThread:1987335 [sender.py:_start_run_threads():1188] run started: 52k33ku2 with start time 1735962323.763129 |
| 2025-01-03 22:45:24,066 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-03 22:45:24,066 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: check_version |
| 2025-01-03 22:45:24,140 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-03 22:45:24,145 DEBUG HandlerThread:1987335 [system_info.py:__init__():26] System info init |
| 2025-01-03 22:45:24,145 DEBUG HandlerThread:1987335 [system_info.py:__init__():41] System info init done |
| 2025-01-03 22:45:24,145 INFO HandlerThread:1987335 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-03 22:45:24,145 INFO SystemMonitor:1987335 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-03 22:45:24,146 INFO HandlerThread:1987335 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-03 22:45:24,146 INFO SystemMonitor:1987335 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-03 22:45:24,146 INFO SystemMonitor:1987335 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-03 22:45:24,147 INFO SystemMonitor:1987335 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-03 22:45:24,148 INFO SystemMonitor:1987335 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-03 22:45:24,149 INFO SystemMonitor:1987335 [interfaces.py:start():188] Started network monitoring |
| 2025-01-03 22:45:24,158 DEBUG HandlerThread:1987335 [system_info.py:probe():152] Probing system |
| 2025-01-03 22:45:24,160 DEBUG HandlerThread:1987335 [system_info.py:_probe_git():137] Probing git |
| 2025-01-03 22:45:24,164 DEBUG HandlerThread:1987335 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-03 22:45:24,165 DEBUG HandlerThread:1987335 [system_info.py:probe():200] Probing system done |
| 2025-01-03 22:45:24,165 DEBUG HandlerThread:1987335 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-04T03:45:24.158820', 'startedAt': '2025-01-04T03:45:23.754665', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'c537eb6734155592809c2a7db5cca971ec186d72'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.738125, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.938, 'min': 800.0, 'max': 4700.0}, {'current': 4499.989, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.063, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 863.9541130065918}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-03 22:45:24,165 INFO HandlerThread:1987335 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-03 22:45:24,165 INFO HandlerThread:1987335 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-03 22:45:24,165 DEBUG HandlerThread:1987335 [system_info.py:_save_code():44] Saving code |
| 2025-01-03 22:45:24,166 DEBUG HandlerThread:1987335 [system_info.py:_save_code():67] Saving code done |
| 2025-01-03 22:45:24,166 DEBUG HandlerThread:1987335 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-03 22:45:24,228 DEBUG HandlerThread:1987335 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-03 22:45:24,228 INFO HandlerThread:1987335 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-03 22:45:24,231 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:45:24,231 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-03 22:45:24,232 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-03 22:45:24,232 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-03 22:45:24,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-03 22:45:24,471 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-03 22:45:24,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:45:24,472 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:45:24,473 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/60cew8al-wandb-metadata.json |
| 2025-01-03 22:45:24,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:24,473 INFO HandlerThread:1987335 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/tb" |
| save: true |
| } |
| control { |
| relay_id: "78a19ede38ec44a88ed85a83a538ee6b" |
| } |
| _info { |
| stream_id: "52k33ku2" |
| } |
|
|
| 2025-01-03 22:45:24,474 DEBUG HandlerThread:1987335 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-03 22:45:24,492 INFO wandb-upload_1:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/9xy9zl4z-code/FISH/eval_robot.py |
| 2025-01-03 22:45:24,500 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/cnnbt49m-diff.patch |
| 2025-01-03 22:45:24,621 DEBUG SenderThread:1987335 [sender.py:send():379] send: telemetry |
| 2025-01-03 22:45:24,621 DEBUG SenderThread:1987335 [sender.py:send():379] send: tbrecord |
| 2025-01-03 22:45:24,622 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:45:24,622 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 with policy live |
| 2025-01-03 22:45:24,850 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/fwjmtloj-events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/code/FISH/eval_robot.py |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/diff.patch |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-metadata.json |
| 2025-01-03 22:45:25,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/requirements.txt |
| 2025-01-03 22:45:25,054 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/code/FISH |
| 2025-01-03 22:45:25,054 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/code |
| 2025-01-03 22:45:25,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:26,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:27,052 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:27,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:28,214 DEBUG SenderThread:1987335 [sender.py:send():379] send: config |
| 2025-01-03 22:45:28,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:29,223 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:29,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:30,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:31,053 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:31,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:32,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:33,054 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:33,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:34,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:34,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:35,055 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:35,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:36,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:37,055 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:37,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:38,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:39,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:45:39,471 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:45:39,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:39,568 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:40,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:41,056 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:41,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:42,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:43,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:44,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:45,069 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:45:45,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:45,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:46,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:47,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:48,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:49,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:50,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:50,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:51,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:52,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:53,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:54,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:45:54,472 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:45:54,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:55,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:55,572 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:45:56,072 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/config.yaml |
| 2025-01-03 22:45:56,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:57,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:58,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:45:59,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:00,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:01,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:01,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:02,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:03,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:04,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:05,478 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:06,480 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:07,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:07,480 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:08,480 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:09,076 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:46:09,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:46:09,471 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:46:09,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:10,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:10,795 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:46:10,795 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 with policy now |
| 2025-01-03 22:46:10,795 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:46:10,796 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:46:10,797 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:46:10,797 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval |
| 2025-01-03 22:46:11,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media |
| 2025-01-03 22:46:11,171 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/rxevysvd-media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 |
| 2025-01-03 22:46:11,429 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/5s9bkhuu-media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 |
| 2025-01-03 22:46:11,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:12,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:12,802 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:13,077 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:46:13,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:14,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:15,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:16,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:17,078 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:46:17,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:18,161 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:18,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:19,079 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:46:19,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:20,481 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:21,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:22,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:23,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:23,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:24,149 DEBUG SystemMonitor:1987335 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-03 22:46:24,149 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:46:24,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:46:24,472 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:46:24,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:25,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:26,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:27,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:28,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:28,529 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:29,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:30,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:31,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:32,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:33,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:34,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:34,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:35,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:36,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:37,483 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:38,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:39,471 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:46:39,472 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:46:39,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:39,619 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:40,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:41,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:42,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:43,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:44,484 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:45,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:45,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:46,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:47,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:48,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:49,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:50,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:50,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:51,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:52,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:53,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:54,150 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:46:54,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:46:54,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:46:54,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:55,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:55,530 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:46:56,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:57,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:58,486 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:46:59,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:00,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:01,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:01,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:02,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:03,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:04,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:05,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:06,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:07,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:07,487 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:08,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:09,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:47:09,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:47:09,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:10,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:11,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:12,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:12,521 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:13,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:14,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:15,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:16,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:17,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:18,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:18,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:19,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:20,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:21,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:22,489 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:23,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:23,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:24,150 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:47:24,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:47:24,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:47:24,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:25,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:26,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:27,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:28,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:28,532 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:29,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:30,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:31,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:32,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:33,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:34,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:34,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:35,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:36,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:37,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:38,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:39,472 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:47:39,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:47:39,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:39,612 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:40,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:41,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:42,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:43,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:44,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:45,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:45,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:46,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:47,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:48,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:49,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:50,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:51,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:51,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:52,493 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:53,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:54,151 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:47:54,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:47:54,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:47:54,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:55,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:56,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:56,558 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:47:57,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:58,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:47:59,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:00,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:01,494 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:02,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:02,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:03,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:04,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:05,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:06,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:07,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:07,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:08,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:09,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:48:09,473 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:48:09,513 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:10,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:11,495 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:12,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:12,624 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:13,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:14,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:15,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:16,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:17,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:18,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:18,496 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:19,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:20,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:21,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:22,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:23,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:24,151 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:48:24,153 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:24,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:48:24,474 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:48:24,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:25,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:26,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:27,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:28,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:29,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:29,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:30,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:31,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:32,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:33,123 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:48:33,363 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/stcnm5gy-events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:48:33,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:34,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:35,124 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:48:35,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:35,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:36,498 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:37,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:38,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:39,125 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:48:39,473 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:48:39,474 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:48:39,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:40,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:40,596 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:41,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:42,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:43,499 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:44,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:45,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:46,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:46,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:47,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:48,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:49,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:50,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:51,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:52,477 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:52,500 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:53,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:54,152 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:48:54,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:48:54,474 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:48:54,521 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:55,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:56,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:57,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:57,541 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:48:58,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:48:59,501 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:00,502 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:01,503 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:02,503 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:03,478 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:03,529 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:04,508 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:04,569 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:49:04,569 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:49:04,569 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 with policy now |
| 2025-01-03 22:49:04,570 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:49:04,570 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:49:04,570 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:49:04,823 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/w0gv9v2t-media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 |
| 2025-01-03 22:49:05,134 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:05,134 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:49:05,134 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 |
| 2025-01-03 22:49:05,134 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval |
| 2025-01-03 22:49:05,399 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/gtthix5c-media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 |
| 2025-01-03 22:49:05,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:06,512 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:07,134 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:07,519 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:08,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:09,135 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:09,261 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:09,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:49:09,474 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:49:09,519 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:10,529 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:11,514 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:12,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:13,137 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:13,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:14,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:14,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:15,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:16,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:17,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:18,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:19,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:19,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:20,515 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:21,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:22,030 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:49:22,030 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:49:22,032 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:49:22,139 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:49:22,140 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:49:22,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:23,140 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:23,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:24,152 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:49:24,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:49:24,475 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:49:24,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:24,611 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:25,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:26,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:27,141 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:27,516 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:28,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:29,142 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:29,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:30,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:30,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:31,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:32,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:33,517 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:34,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:35,490 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:35,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:36,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:37,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:38,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:39,474 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:49:39,474 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:49:39,521 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:40,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:40,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:41,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:42,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:43,146 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:43,518 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:44,519 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:45,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:46,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:46,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:47,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:48,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:49,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:50,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:51,491 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:51,525 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:52,527 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:53,150 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:53,326 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:49:53,326 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 with policy now |
| 2025-01-03 22:49:53,330 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:49:53,330 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:49:53,330 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:49:53,331 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:49:53,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:53,598 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/fa955bxj-media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 |
| 2025-01-03 22:49:54,150 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:49:54,150 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 |
| 2025-01-03 22:49:54,150 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval |
| 2025-01-03 22:49:54,153 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:49:54,430 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/cu5o6l11-media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 |
| 2025-01-03 22:49:54,485 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:49:54,486 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:49:54,531 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:55,150 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:55,533 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:56,534 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:56,627 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:49:57,528 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:58,529 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:49:59,152 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:49:59,533 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:00,530 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:01,152 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:01,533 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:01,805 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:02,534 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:03,549 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:03,858 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:50:03,858 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:50:03,859 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:50:04,153 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:50:04,154 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:50:04,381 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/o7vv2xe6-events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:50:04,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:05,154 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:05,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:06,538 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:07,155 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:07,432 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:07,534 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:08,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:09,155 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:09,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:50:09,475 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:50:09,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:10,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:11,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:12,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:12,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:13,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:14,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:15,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:16,535 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:17,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:18,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:18,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:19,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:20,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:21,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:22,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:23,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:24,154 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:50:24,154 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:24,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:50:24,476 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:50:24,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:25,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:26,536 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:27,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:28,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:29,492 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:29,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:30,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:31,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:32,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:33,537 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:34,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:35,164 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:35,218 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:35,549 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:35,726 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:50:35,726 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 with policy now |
| 2025-01-03 22:50:35,728 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:50:35,729 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:50:35,729 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:50:35,729 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:50:35,973 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/ux2eefwg-media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 |
| 2025-01-03 22:50:36,164 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:50:36,164 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 |
| 2025-01-03 22:50:36,165 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval |
| 2025-01-03 22:50:36,431 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/vvr8yt8y-media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 |
| 2025-01-03 22:50:36,558 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:37,165 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:37,550 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:38,550 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:39,488 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:50:39,488 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:50:39,550 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:40,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:40,607 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:41,166 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:41,561 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:42,551 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:43,167 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:43,551 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:44,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:45,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:46,497 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:46,556 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:47,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:48,168 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:50:48,565 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:48,915 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:50:48,915 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:50:48,916 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:50:49,169 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:49,169 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:50:49,561 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:50,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:51,169 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:51,508 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:51,563 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:52,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:53,170 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:50:53,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:54,154 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:50:54,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:50:54,476 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:50:54,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:55,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:56,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:56,563 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:50:57,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:58,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:50:59,557 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:00,558 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:01,173 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:01,558 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:01,801 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:02,558 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:03,174 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:03,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:04,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:05,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:06,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:07,503 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:07,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:08,559 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:09,475 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:51:09,476 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:51:09,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:10,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:11,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:12,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:13,178 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:13,217 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:13,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:14,560 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:15,562 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:16,561 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:17,561 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:18,503 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:18,561 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:19,564 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:20,103 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:51:20,103 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 with policy now |
| 2025-01-03 22:51:20,105 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:51:20,106 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:51:20,106 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:20,107 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:20,180 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:20,180 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 |
| 2025-01-03 22:51:20,181 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval |
| 2025-01-03 22:51:20,357 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/5o0r58z3-media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 |
| 2025-01-03 22:51:20,567 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:20,598 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/fu648ifn-media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 |
| 2025-01-03 22:51:21,181 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:21,570 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:22,577 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:23,574 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:23,752 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:24,155 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:51:24,482 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:51:24,482 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:51:24,579 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:25,182 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:25,589 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:26,594 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:27,592 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:28,594 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:29,183 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:29,522 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:29,593 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:30,591 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:31,597 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:32,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:33,600 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:34,522 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:34,597 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:35,610 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:35,980 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:51:35,981 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:35,981 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:36,186 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:36,186 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:51:36,258 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 22:51:36,258 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 22:51:36,418 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:51:36,437 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/q3ju93tp-events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:51:36,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:36,739 INFO wandb-upload_0:1987335 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpq61gnfh2 |
| 2025-01-03 22:51:36,838 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 22:51:36,846 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 22:51:37,098 INFO SenderThread:1987335 [sender.py:send_request_log_artifact():1525] logged artifact run-52k33ku2-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzk5NzUwOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY5MTcyMQ==', 'latestArtifact': None}} |
| 2025-01-03 22:51:37,098 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:51:37,098 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-03 22:51:37,099 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:51:37,099 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,099 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,099 DEBUG SenderThread:1987335 [sender.py:send():379] send: telemetry |
| 2025-01-03 22:51:37,100 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,100 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,100 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,100 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,101 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,102 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,102 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,103 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,103 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,103 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,103 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,104 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,104 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,104 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,104 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,105 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,105 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,105 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,105 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,106 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,106 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,106 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:37,106 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 22:51:37,186 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:37,186 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table/eval |
| 2025-01-03 22:51:37,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table |
| 2025-01-03 22:51:37,287 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/4nd68nxt-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-03 22:51:37,416 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/hg3nn0ra-media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json |
| 2025-01-03 22:51:37,495 INFO wandb-upload_1:1987335 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmph62jfnpw |
| 2025-01-03 22:51:37,527 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/osfv52tg-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-03 22:51:37,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:37,861 INFO SenderThread:1987335 [sender.py:send_request_log_artifact():1525] logged artifact run-52k33ku2-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzk5NzUyMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY5MTcyMg==', 'latestArtifact': None}} |
| 2025-01-03 22:51:37,861 DEBUG SenderThread:1987335 [sender.py:send():379] send: files |
| 2025-01-03 22:51:37,861 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json with policy now |
| 2025-01-03 22:51:37,862 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:51:37,862 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:37,862 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:38,108 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/y5r6l74i-media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json |
| 2025-01-03 22:51:38,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:38,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:39,187 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:39,476 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 22:51:39,476 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 22:51:39,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:39,627 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:40,599 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:41,600 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:42,621 DEBUG SenderThread:1987335 [sender.py:send():379] send: exit |
| 2025-01-03 22:51:42,621 INFO SenderThread:1987335 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-03 22:51:42,622 INFO SenderThread:1987335 [sender.py:send_exit():588] handling runtime: 378 |
| 2025-01-03 22:51:42,622 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:42,622 INFO SenderThread:1987335 [sender.py:send_exit():594] send defer |
| 2025-01-03 22:51:42,622 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:42,622 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-03 22:51:42,622 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:42,622 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-03 22:51:42,622 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-03 22:51:42,622 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:42,623 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-03 22:51:42,623 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:42,623 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-03 22:51:42,623 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-03 22:51:42,623 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:42,623 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-03 22:51:42,623 INFO HandlerThread:1987335 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-03 22:51:42,623 DEBUG SystemMonitor:1987335 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-03 22:51:42,623 DEBUG SystemMonitor:1987335 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-03 22:51:42,624 INFO HandlerThread:1987335 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-03 22:51:42,624 INFO HandlerThread:1987335 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-03 22:51:42,676 INFO HandlerThread:1987335 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-03 22:51:42,676 INFO HandlerThread:1987335 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-03 22:51:42,676 INFO HandlerThread:1987335 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-03 22:51:42,676 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:42,677 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-03 22:51:42,677 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-03 22:51:42,677 DEBUG SenderThread:1987335 [sender.py:send():379] send: stats |
| 2025-01-03 22:51:42,677 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:42,677 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-03 22:51:42,677 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:42,677 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-03 22:51:42,677 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-03 22:51:42,677 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:42,677 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-03 22:51:43,189 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:45,190 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:48,987 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:48,987 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:48,987 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-03 22:51:48,988 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:48,988 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-03 22:51:48,988 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-03 22:51:48,988 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:48,988 DEBUG SenderThread:1987335 [sender.py:send():379] send: history |
| 2025-01-03 22:51:48,988 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 22:51:48,989 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:48,989 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-03 22:51:48,989 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:48,989 DEBUG SenderThread:1987335 [sender.py:send():379] send: summary |
| 2025-01-03 22:51:48,993 INFO SenderThread:1987335 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 22:51:48,993 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:48,994 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-03 22:51:48,994 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-03 22:51:48,994 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:48,994 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-03 22:51:48,994 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:48,994 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-03 22:51:48,997 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 22:51:49,171 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-03 22:51:49,171 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:49,171 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-03 22:51:49,171 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:49,171 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-03 22:51:49,191 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/config.yaml |
| 2025-01-03 22:51:49,191 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:49,622 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:51,189 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-03 22:51:51,189 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:51,189 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:51,190 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-03 22:51:51,190 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:51,190 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-03 22:51:51,190 INFO SenderThread:1987335 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-03 22:51:51,192 INFO SenderThread:1987335 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-03 22:51:51,195 INFO Thread-12 :1987335 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:51,195 INFO SenderThread:1987335 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-03 22:51:51,205 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-03 22:51:51,205 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:51,205 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-03 22:51:51,206 DEBUG SenderThread:1987335 [sender.py:send():379] send: artifact |
| 2025-01-03 22:51:51,435 INFO SenderThread:1987335 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNzk5MDI2MQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNzk5MDI2MQ==', 'versionIndex': 413}}} |
| 2025-01-03 22:51:51,435 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:51,435 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-03 22:51:51,435 INFO SenderThread:1987335 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-03 22:51:51,623 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:52,195 INFO SenderThread:1987335 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/config.yaml config.yaml |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json wandb-summary.json |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/requirements.txt requirements.txt |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:51:52,196 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log output.log |
| 2025-01-03 22:51:52,199 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/diff.patch diff.patch |
| 2025-01-03 22:51:52,201 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 media/videos/eval/2_eval_2_df0f4e50e7a3af14dc5d.mp4 |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 media/videos/eval/0_eval_0_f8807ddb3aac3464696f.mp4 |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 media/videos/eval/4_eval_4_2aa04d437225714191e3.mp4 |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 media/videos/eval/1_eval_1_93cac9f740e166267128.mp4 |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 media/videos/eval/3_eval_3_c01cf859d0173b75d00e.mp4 |
| 2025-01-03 22:51:52,202 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json media/table/eval/total_success_rate_ci_6_30dbc0829a5787691438.table.json |
| 2025-01-03 22:51:52,204 INFO SenderThread:1987335 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-03 22:51:52,204 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-03 22:51:52,204 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:52,204 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:52,204 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-03 22:51:52,206 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:52,206 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-03 22:51:52,206 INFO SenderThread:1987335 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 22:51:52,378 INFO wandb-upload_0:1987335 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/requirements.txt |
| 2025-01-03 22:51:52,624 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:52,624 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:53,325 INFO wandb-upload_1:1987335 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/wandb-summary.json |
| 2025-01-03 22:51:53,326 INFO wandb-upload_2:1987335 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/config.yaml |
| 2025-01-03 22:51:53,368 INFO wandb-upload_3:1987335 [upload_job.py:push():130] Uploaded file /tmp/tmpb9z9_5tzwandb/kfym2c35-events.out.tfevents.1735962324.leonmkim-ROG-Strix-G15CS-G15CS.1987170.0 |
| 2025-01-03 22:51:53,375 INFO wandb-upload_4:1987335 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/files/output.log |
| 2025-01-03 22:51:53,575 INFO Thread-11 (_thread_body):1987335 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-03 22:51:53,575 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:53,575 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-03 22:51:53,576 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:53,576 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-03 22:51:53,576 INFO SenderThread:1987335 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-03 22:51:53,577 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-03 22:51:53,577 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:53,577 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-03 22:51:53,577 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:53,577 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-03 22:51:53,578 INFO SenderThread:1987335 [file_stream.py:finish():601] file stream finish called |
| 2025-01-03 22:51:53,624 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:53,645 INFO SenderThread:1987335 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-03 22:51:53,646 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-03 22:51:53,646 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:53,646 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:53,646 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-03 22:51:53,646 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:53,646 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-03 22:51:53,646 INFO SenderThread:1987335 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-03 22:51:53,646 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 22:51:53,646 INFO HandlerThread:1987335 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-03 22:51:53,647 DEBUG SenderThread:1987335 [sender.py:send():379] send: final |
| 2025-01-03 22:51:53,647 DEBUG SenderThread:1987335 [sender.py:send():379] send: footer |
| 2025-01-03 22:51:53,647 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 22:51:53,647 INFO SenderThread:1987335 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-03 22:51:53,647 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:53,647 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-03 22:51:53,647 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 22:51:53,648 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-03 22:51:53,648 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:53,648 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-03 22:51:53,649 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: server_info |
| 2025-01-03 22:51:53,649 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 22:51:53,713 DEBUG SenderThread:1987335 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 22:51:53,713 INFO MainThread:1987335 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-03 22:51:53,713 INFO MainThread:1987335 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-03 22:51:53,714 INFO MainThread:1987335 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-03 22:51:53,714 DEBUG HandlerThread:1987335 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-03 22:51:53,714 INFO HandlerThread:1987335 [handler.py:finish():882] shutting down handler |
| 2025-01-03 22:51:54,648 INFO WriterThread:1987335 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/52k33ku2/wandb/run-20250103_224523-52k33ku2/run-52k33ku2.wandb |
| 2025-01-03 22:51:54,713 INFO SenderThread:1987335 [sender.py:finish():1615] shutting down sender |
| 2025-01-03 22:51:54,713 INFO SenderThread:1987335 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 22:51:54,713 INFO SenderThread:1987335 [file_pusher.py:join():175] waiting for file pusher |
|
|
