| 2025-01-06 22:13:22,748 INFO StreamThr :2122155 [internal.py:wandb_internal():85] W&B internal server running at pid: 2122155, started at: 2025-01-06 22:13:22.748336 |
| 2025-01-06 22:13:22,750 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 22:13:22,763 INFO WriterThread:2122155 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/run-9ig9megf.wandb |
| 2025-01-06 22:13:22,763 DEBUG SenderThread:2122155 [sender.py:send():379] send: header |
| 2025-01-06 22:13:22,771 DEBUG SenderThread:2122155 [sender.py:send():379] send: run |
| 2025-01-06 22:13:23,035 INFO SenderThread:2122155 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files |
| 2025-01-06 22:13:23,035 INFO SenderThread:2122155 [sender.py:_start_run_threads():1188] run started: 9ig9megf with start time 1736219602.749401 |
| 2025-01-06 22:13:23,054 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 22:13:23,054 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 22:13:23,105 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 22:13:23,110 DEBUG HandlerThread:2122155 [system_info.py:__init__():26] System info init |
| 2025-01-06 22:13:23,110 DEBUG HandlerThread:2122155 [system_info.py:__init__():41] System info init done |
| 2025-01-06 22:13:23,111 INFO HandlerThread:2122155 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 22:13:23,111 INFO SystemMonitor:2122155 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 22:13:23,111 INFO HandlerThread:2122155 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 22:13:23,111 INFO SystemMonitor:2122155 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 22:13:23,112 INFO SystemMonitor:2122155 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 22:13:23,112 INFO SystemMonitor:2122155 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 22:13:23,113 INFO SystemMonitor:2122155 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 22:13:23,113 INFO SystemMonitor:2122155 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 22:13:23,124 DEBUG HandlerThread:2122155 [system_info.py:probe():152] Probing system |
| 2025-01-06 22:13:23,125 DEBUG HandlerThread:2122155 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 22:13:23,130 DEBUG HandlerThread:2122155 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 22:13:23,130 DEBUG HandlerThread:2122155 [system_info.py:probe():200] Probing system done |
| 2025-01-06 22:13:23,130 DEBUG HandlerThread:2122155 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T03:13:23.124287', 'startedAt': '2025-01-07T03:13:22.738878', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3748.5340000000006, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4491.899, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.29, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.995, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 558.2445449829102}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 22:13:23,130 INFO HandlerThread:2122155 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 22:13:23,130 INFO HandlerThread:2122155 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 22:13:23,130 DEBUG HandlerThread:2122155 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 22:13:23,131 DEBUG HandlerThread:2122155 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 22:13:23,132 DEBUG HandlerThread:2122155 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 22:13:23,192 DEBUG HandlerThread:2122155 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 22:13:23,193 INFO HandlerThread:2122155 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 22:13:23,196 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:13:23,196 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 22:13:23,196 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 22:13:23,197 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 22:13:23,440 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/812ddovu-wandb-metadata.json |
| 2025-01-06 22:13:23,464 INFO wandb-upload_2:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/8bsaca9a-diff.patch |
| 2025-01-06 22:13:23,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 22:13:23,467 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 22:13:23,468 DEBUG SenderThread:2122155 [sender.py:send():379] send: telemetry |
| 2025-01-06 22:13:23,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:13:23,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:13:23,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:23,470 INFO HandlerThread:2122155 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/tb" |
| save: true |
| } |
| control { |
| relay_id: "d08939d596444ee6b2c634f165868670" |
| } |
| _info { |
| stream_id: "9ig9megf" |
| } |
|
|
| 2025-01-06 22:13:23,470 DEBUG HandlerThread:2122155 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 22:13:23,488 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/oh5ciig3-code/FISH/eval_robot.py |
| 2025-01-06 22:13:23,570 DEBUG SenderThread:2122155 [sender.py:send():379] send: tbrecord |
| 2025-01-06 22:13:23,571 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:13:23,571 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 with policy live |
| 2025-01-06 22:13:23,758 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/ig0mgqma-events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:13:24,036 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/code/FISH/eval_robot.py |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/requirements.txt |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-metadata.json |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/diff.patch |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/code |
| 2025-01-06 22:13:24,037 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/code/FISH |
| 2025-01-06 22:13:24,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:25,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:26,036 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:26,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:27,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:28,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:28,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:29,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:30,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:31,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:32,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:33,334 DEBUG SenderThread:2122155 [sender.py:send():379] send: config |
| 2025-01-06 22:13:33,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:33,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:34,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:35,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:36,039 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:36,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:37,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:38,040 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:38,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:13:38,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:13:38,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:38,535 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:39,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:40,040 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:40,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:41,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:42,041 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:13:42,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:43,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:44,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:44,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:45,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:46,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:47,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:48,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:49,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:49,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:50,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:51,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:52,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:53,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:13:53,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:13:53,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:54,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:54,563 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:13:55,044 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/config.yaml |
| 2025-01-06 22:13:55,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:56,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:57,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:58,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:13:59,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:00,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:00,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:01,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:02,046 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:02,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:03,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:04,047 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:04,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:05,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:05,888 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:06,048 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:06,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:07,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:08,467 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:14:08,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:14:08,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:09,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:10,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:11,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:11,581 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:12,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:13,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:14,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:15,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:16,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:17,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:17,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:18,474 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:19,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:20,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:21,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:22,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:23,114 DEBUG SystemMonitor:2122155 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 22:14:23,114 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:14:23,115 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:23,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:14:23,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:14:23,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:24,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:25,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:26,476 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:27,475 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:28,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:28,476 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:29,476 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:30,476 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:31,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:32,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:32,601 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:14:32,602 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:14:32,602 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 with policy now |
| 2025-01-06 22:14:32,602 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:14:32,602 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:14:32,603 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:14:32,859 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/lil8d117-media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 |
| 2025-01-06 22:14:33,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 |
| 2025-01-06 22:14:33,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:14:33,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media |
| 2025-01-06 22:14:33,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval |
| 2025-01-06 22:14:33,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos |
| 2025-01-06 22:14:33,331 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/41hp5crh-media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 |
| 2025-01-06 22:14:33,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:33,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:34,055 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:34,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:35,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:36,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:37,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:38,056 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:38,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:14:38,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:14:38,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:38,539 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:39,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:40,057 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:40,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:41,478 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:42,478 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:43,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:44,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:44,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:45,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:46,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:47,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:48,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:49,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:50,061 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:50,061 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:14:50,087 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:50,304 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/85bojlbt-events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:14:50,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:51,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:52,061 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:52,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:53,115 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:14:53,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:14:53,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:14:53,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:54,062 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:14:54,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:55,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:55,527 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:14:56,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:57,480 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:58,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:14:59,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:00,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:00,768 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:01,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:02,065 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:02,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:03,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:04,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:05,481 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:06,413 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:06,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:07,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:08,067 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:08,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:15:08,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:15:08,513 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:09,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:10,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:11,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:11,593 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:12,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:13,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:14,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:15,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:16,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:17,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:17,482 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:18,483 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:19,483 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:20,483 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:21,174 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:15:21,174 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 with policy now |
| 2025-01-06 22:15:21,178 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:15:21,179 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:15:21,179 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:15:21,179 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:15:21,426 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/ty8szote-media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 |
| 2025-01-06 22:15:21,485 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:22,071 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:22,071 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:15:22,071 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 |
| 2025-01-06 22:15:22,072 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval |
| 2025-01-06 22:15:22,426 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/9vjyf2vq-media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 |
| 2025-01-06 22:15:22,484 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:23,115 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:15:23,115 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:23,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:15:23,469 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:15:23,513 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:24,072 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:24,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:25,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:26,072 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:26,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:27,486 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:28,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:28,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:29,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:30,074 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:30,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:31,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:31,932 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:15:31,932 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:15:31,932 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:15:32,075 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:15:32,075 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:15:32,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:33,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:33,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:34,075 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:34,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:35,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:36,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:37,487 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:38,077 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:38,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:15:38,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:15:38,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:38,535 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:39,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:40,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:41,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:42,078 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:42,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:43,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:44,079 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:15:44,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:44,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:45,488 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:46,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:47,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:48,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:49,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:49,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:50,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:51,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:52,489 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:53,116 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:15:53,468 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:15:53,468 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:15:53,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:54,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:54,601 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:15:55,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:56,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:57,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:58,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:15:59,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:00,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:00,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:01,490 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:02,491 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:03,375 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:16:03,375 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 with policy now |
| 2025-01-06 22:16:03,377 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:16:03,378 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:16:03,378 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:16:03,379 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:16:03,491 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:03,671 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/2m9ytad9-media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 |
| 2025-01-06 22:16:04,087 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:04,087 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:16:04,087 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 |
| 2025-01-06 22:16:04,087 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval |
| 2025-01-06 22:16:04,385 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/p42gscn4-media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 |
| 2025-01-06 22:16:04,492 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:05,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:05,497 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:06,087 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:06,500 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:07,501 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:08,088 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:08,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:16:08,471 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:16:08,513 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:09,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:10,505 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:10,781 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:11,506 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:11,757 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:16:11,757 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:16:11,775 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:16:12,090 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:12,090 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:16:12,090 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:16:12,329 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/832f55tw-events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:16:12,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:13,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:14,090 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:14,512 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:15,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:16,003 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:16,091 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:16,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:17,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:18,092 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:18,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:19,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:20,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:21,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:21,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:22,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:23,116 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:16:23,469 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:16:23,470 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:16:23,513 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:24,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:25,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:26,507 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:26,599 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:27,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:28,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:29,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:30,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:31,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:32,472 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:32,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:33,508 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:34,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:35,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:36,098 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:36,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:37,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:38,229 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:38,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:16:38,470 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:16:38,517 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:39,510 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:40,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:41,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:42,509 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:43,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:43,516 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:44,057 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:16:44,057 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 with policy now |
| 2025-01-06 22:16:44,061 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:16:44,061 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:16:44,061 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:16:44,062 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:16:44,100 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:44,101 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:16:44,101 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 |
| 2025-01-06 22:16:44,101 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval |
| 2025-01-06 22:16:44,312 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/6xeimi59-media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 |
| 2025-01-06 22:16:44,514 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:44,592 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/3xd02x84-media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 |
| 2025-01-06 22:16:45,517 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:46,101 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:46,514 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:47,518 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:48,519 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:49,477 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:49,523 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:50,102 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:50,519 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:51,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:52,103 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:52,520 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:53,117 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:16:53,473 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:16:53,473 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:16:53,523 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:54,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:54,539 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:16:55,523 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:56,104 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:16:56,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:56,813 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:16:56,813 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:16:56,813 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:16:57,105 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:16:57,526 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:58,105 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:16:58,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:59,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:16:59,762 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:00,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:01,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:02,107 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:02,524 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:03,528 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:04,528 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:05,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:05,528 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:06,528 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:07,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:08,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:17:08,470 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:17:08,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:09,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:10,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:10,536 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:11,529 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:12,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:13,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:14,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:15,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:16,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:16,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:17,530 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:18,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:19,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:20,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:21,479 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:21,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:22,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:23,117 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:17:23,470 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:17:23,470 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:17:23,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:24,531 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:25,532 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:26,532 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:26,539 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:27,536 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:28,118 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:28,333 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:17:28,333 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 with policy now |
| 2025-01-06 22:17:28,337 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:17:28,338 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:17:28,338 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:28,338 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:28,532 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:28,585 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/17fgcz54-media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 |
| 2025-01-06 22:17:29,119 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:17:29,119 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 |
| 2025-01-06 22:17:29,119 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval |
| 2025-01-06 22:17:29,409 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/uh1fc3so-media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 |
| 2025-01-06 22:17:29,534 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:30,119 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:30,537 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:31,542 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:31,963 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:32,544 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:33,551 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:34,120 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:34,548 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:35,546 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:36,121 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:36,552 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:37,320 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:37,550 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:37,775 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 22:17:37,775 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 22:17:37,947 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:17:38,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:38,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:17:38,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-06 22:17:38,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media |
| 2025-01-06 22:17:38,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table |
| 2025-01-06 22:17:38,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval |
| 2025-01-06 22:17:38,336 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/k02egdlf-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-06 22:17:38,346 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 22:17:38,355 INFO wandb-upload_2:2122155 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmplup4j1go |
| 2025-01-06 22:17:38,358 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:17:38,360 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/gylawia1-events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:17:38,471 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:17:38,547 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:38,744 INFO SenderThread:2122155 [sender.py:send_request_log_artifact():1525] logged artifact run-9ig9megf-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjU2MDY2OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDczNTY1Nw==', 'latestArtifact': None}} |
| 2025-01-06 22:17:38,744 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:17:38,744 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now |
| 2025-01-06 22:17:38,745 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:17:38,745 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,745 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,746 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:17:38,746 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,749 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,749 DEBUG SenderThread:2122155 [sender.py:send():379] send: telemetry |
| 2025-01-06 22:17:38,749 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,750 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,750 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,750 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,750 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,751 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,751 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,751 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,751 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,752 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,752 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,752 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,752 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,753 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,753 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,753 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,753 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,754 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,754 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,754 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,754 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:38,755 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:38,755 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 22:17:38,991 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/oa86qmqk-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-06 22:17:39,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:17:39,122 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:17:39,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json |
| 2025-01-06 22:17:39,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval |
| 2025-01-06 22:17:39,132 INFO wandb-upload_1:2122155 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpeqtnnbg3 |
| 2025-01-06 22:17:39,306 INFO wandb-upload_2:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/m76450mq-media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json |
| 2025-01-06 22:17:39,535 INFO SenderThread:2122155 [sender.py:send_request_log_artifact():1525] logged artifact run-9ig9megf-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjU2MDY4MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDczNTY2MA==', 'latestArtifact': None}} |
| 2025-01-06 22:17:39,535 DEBUG SenderThread:2122155 [sender.py:send():379] send: files |
| 2025-01-06 22:17:39,536 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json with policy now |
| 2025-01-06 22:17:39,536 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:17:39,536 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:39,537 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:39,537 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:17:39,547 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:39,741 DEBUG SenderThread:2122155 [sender.py:send():379] send: exit |
| 2025-01-06 22:17:39,742 INFO SenderThread:2122155 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 22:17:39,742 INFO SenderThread:2122155 [sender.py:send_exit():588] handling runtime: 256 |
| 2025-01-06 22:17:39,742 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:39,742 INFO SenderThread:2122155 [sender.py:send_exit():594] send defer |
| 2025-01-06 22:17:39,742 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:39,742 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 22:17:39,743 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:39,743 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 22:17:39,743 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 22:17:39,743 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:39,743 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 22:17:39,743 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:39,743 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 22:17:39,743 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 22:17:39,743 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:39,743 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 22:17:39,743 INFO HandlerThread:2122155 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 22:17:39,744 INFO HandlerThread:2122155 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 22:17:39,744 INFO HandlerThread:2122155 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 22:17:39,744 DEBUG SystemMonitor:2122155 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 22:17:39,744 DEBUG SystemMonitor:2122155 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 22:17:39,780 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/gmfg8mam-media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json |
| 2025-01-06 22:17:39,794 INFO HandlerThread:2122155 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 22:17:39,794 INFO HandlerThread:2122155 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 22:17:39,794 INFO HandlerThread:2122155 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 22:17:39,795 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:39,795 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 22:17:39,795 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 22:17:39,795 DEBUG SenderThread:2122155 [sender.py:send():379] send: stats |
| 2025-01-06 22:17:39,795 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:39,795 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 22:17:39,795 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:39,795 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 22:17:39,795 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 22:17:39,796 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:39,796 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 22:17:40,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:40,123 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:17:42,124 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:43,124 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/config.yaml |
| 2025-01-06 22:17:45,871 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:45,871 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:45,871 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 22:17:45,872 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:45,872 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 22:17:45,872 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 22:17:45,872 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:45,872 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:45,872 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 22:17:45,872 DEBUG SenderThread:2122155 [sender.py:send():379] send: history |
| 2025-01-06 22:17:45,873 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:17:45,876 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:45,876 DEBUG SenderThread:2122155 [sender.py:send():379] send: summary |
| 2025-01-06 22:17:45,876 INFO SenderThread:2122155 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:17:45,876 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:45,876 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 22:17:45,876 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 22:17:45,876 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:45,877 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 22:17:45,877 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:45,877 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 22:17:45,877 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 22:17:45,877 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:17:45,877 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:45,877 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 22:17:45,877 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:45,877 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 22:17:46,125 INFO Thread-12 :2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:17:46,743 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:49,237 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 22:17:49,237 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:49,237 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:49,237 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 22:17:49,238 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:49,238 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 22:17:49,238 INFO SenderThread:2122155 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 22:17:49,240 INFO SenderThread:2122155 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 22:17:49,242 INFO SenderThread:2122155 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 22:17:49,256 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 22:17:49,257 DEBUG SenderThread:2122155 [sender.py:send():379] send: artifact |
| 2025-01-06 22:17:49,257 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:49,259 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 22:17:49,503 INFO SenderThread:2122155 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkyMTY1Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} |
| 2025-01-06 22:17:49,503 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:49,503 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 22:17:49,503 INFO SenderThread:2122155 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 22:17:49,744 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:50,126 INFO SenderThread:2122155 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:50,127 INFO SenderThread:2122155 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files |
| 2025-01-06 22:17:50,127 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/config.yaml config.yaml |
| 2025-01-06 22:17:50,127 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 22:17:50,127 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/requirements.txt requirements.txt |
| 2025-01-06 22:17:50,130 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 22:17:50,130 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log output.log |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/diff.patch diff.patch |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 media/videos/eval/0_eval_0_44f7c849dd38a4c00ad4.mp4 |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 media/videos/eval/1_eval_1_1c32d361c2cac363e3ab.mp4 |
| 2025-01-06 22:17:50,132 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 media/videos/eval/2_eval_2_4214c17fed8f2dd3c5b4.mp4 |
| 2025-01-06 22:17:50,133 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 media/videos/eval/3_eval_3_2a87ee96aee6886a00c6.mp4 |
| 2025-01-06 22:17:50,133 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 media/videos/eval/4_eval_4_eed30eb0a971980a2bf8.mp4 |
| 2025-01-06 22:17:50,133 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-06 22:17:50,133 INFO SenderThread:2122155 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json media/table/eval/total_success_rate_ci_6_c696f5811b0274780cec.table.json |
| 2025-01-06 22:17:50,134 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 22:17:50,134 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:50,135 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:50,136 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 22:17:50,136 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:50,136 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 22:17:50,136 INFO SenderThread:2122155 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 22:17:50,303 INFO wandb-upload_2:2122155 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/config.yaml |
| 2025-01-06 22:17:50,312 INFO wandb-upload_1:2122155 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/wandb-summary.json |
| 2025-01-06 22:17:50,320 INFO wandb-upload_0:2122155 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/requirements.txt |
| 2025-01-06 22:17:50,358 INFO wandb-upload_4:2122155 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/files/output.log |
| 2025-01-06 22:17:50,413 INFO wandb-upload_3:2122155 [upload_job.py:push():130] Uploaded file /tmp/tmpza8aoq14wandb/6f9yw2hd-events.out.tfevents.1736219603.leonmkim-ROG-Strix-G15CS-G15CS.2121985.0 |
| 2025-01-06 22:17:50,613 INFO Thread-11 (_thread_body):2122155 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 22:17:50,614 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:50,614 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 22:17:50,614 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:50,614 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 22:17:50,614 INFO SenderThread:2122155 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 22:17:50,615 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 22:17:50,615 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:50,615 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 22:17:50,615 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:50,616 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 22:17:50,616 INFO SenderThread:2122155 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 22:17:50,744 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:50,802 INFO SenderThread:2122155 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 22:17:50,802 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 22:17:50,803 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:50,803 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:50,803 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 22:17:50,803 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:50,803 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 22:17:50,803 INFO SenderThread:2122155 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 22:17:50,803 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:17:50,803 INFO HandlerThread:2122155 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 22:17:50,804 DEBUG SenderThread:2122155 [sender.py:send():379] send: final |
| 2025-01-06 22:17:50,804 DEBUG SenderThread:2122155 [sender.py:send():379] send: footer |
| 2025-01-06 22:17:50,804 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:17:50,804 INFO SenderThread:2122155 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 22:17:50,804 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:50,805 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:50,805 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 22:17:50,805 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:17:50,805 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 22:17:50,805 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 22:17:50,806 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 22:17:50,808 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:17:50,858 DEBUG SenderThread:2122155 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:17:50,858 INFO MainThread:2122155 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 22:17:50,858 INFO MainThread:2122155 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 22:17:50,859 INFO MainThread:2122155 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 22:17:50,859 DEBUG HandlerThread:2122155 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 22:17:50,859 INFO HandlerThread:2122155 [handler.py:finish():882] shutting down handler |
| 2025-01-06 22:17:51,806 INFO WriterThread:2122155 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/9ig9megf/wandb/run-20250106_221322-9ig9megf/run-9ig9megf.wandb |
| 2025-01-06 22:17:51,858 INFO SenderThread:2122155 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 22:17:51,858 INFO SenderThread:2122155 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 22:17:51,858 INFO SenderThread:2122155 [file_pusher.py:join():175] waiting for file pusher |
|
|
