| 2025-01-21 20:56:09,991 INFO StreamThr :2643314 [internal.py:wandb_internal():85] W&B internal server running at pid: 2643314, started at: 2025-01-21 20:56:09.990696 |
| 2025-01-21 20:56:09,992 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status |
| 2025-01-21 20:56:10,004 INFO WriterThread:2643314 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/run-37t5ymwr.wandb |
| 2025-01-21 20:56:10,004 DEBUG SenderThread:2643314 [sender.py:send():379] send: header |
| 2025-01-21 20:56:10,012 DEBUG SenderThread:2643314 [sender.py:send():379] send: run |
| 2025-01-21 20:56:10,256 INFO SenderThread:2643314 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files |
| 2025-01-21 20:56:10,256 INFO SenderThread:2643314 [sender.py:_start_run_threads():1188] run started: 37t5ymwr with start time 1737510969.993125 |
| 2025-01-21 20:56:10,272 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-21 20:56:10,272 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: check_version |
| 2025-01-21 20:56:10,346 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-21 20:56:10,351 DEBUG HandlerThread:2643314 [system_info.py:__init__():26] System info init |
| 2025-01-21 20:56:10,351 DEBUG HandlerThread:2643314 [system_info.py:__init__():41] System info init done |
| 2025-01-21 20:56:10,351 INFO HandlerThread:2643314 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-21 20:56:10,351 INFO SystemMonitor:2643314 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-21 20:56:10,351 INFO HandlerThread:2643314 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-21 20:56:10,352 INFO SystemMonitor:2643314 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-21 20:56:10,352 INFO SystemMonitor:2643314 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-21 20:56:10,353 INFO SystemMonitor:2643314 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-21 20:56:10,353 INFO SystemMonitor:2643314 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-21 20:56:10,354 INFO SystemMonitor:2643314 [interfaces.py:start():188] Started network monitoring |
| 2025-01-21 20:56:10,366 DEBUG HandlerThread:2643314 [system_info.py:probe():152] Probing system |
| 2025-01-21 20:56:10,367 DEBUG HandlerThread:2643314 [system_info.py:_probe_git():137] Probing git |
| 2025-01-21 20:56:10,372 DEBUG HandlerThread:2643314 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-21 20:56:10,372 DEBUG HandlerThread:2643314 [system_info.py:probe():200] Probing system done |
| 2025-01-21 20:56:10,372 DEBUG HandlerThread:2643314 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-22T01:56:10.366323', 'startedAt': '2025-01-22T01:56:09.983032', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'dc9da9fa0ef2996c7fec8d105e7933e728cd952f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3748.394875, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.064, 'min': 800.0, 'max': 4700.0}, {'current': 4499.927, 'min': 800.0, 'max': 4700.0}, {'current': 4499.996, 'min': 800.0, 'max': 4700.0}, {'current': 4494.124, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.596, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.005, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 328.9591178894043}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-21 20:56:10,373 INFO HandlerThread:2643314 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-21 20:56:10,373 INFO HandlerThread:2643314 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-21 20:56:10,373 DEBUG HandlerThread:2643314 [system_info.py:_save_code():44] Saving code |
| 2025-01-21 20:56:10,374 DEBUG HandlerThread:2643314 [system_info.py:_save_code():67] Saving code done |
| 2025-01-21 20:56:10,374 DEBUG HandlerThread:2643314 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-21 20:56:10,437 DEBUG HandlerThread:2643314 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-21 20:56:10,438 INFO HandlerThread:2643314 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-21 20:56:10,441 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:56:10,441 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-21 20:56:10,441 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-21 20:56:10,441 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-21 20:56:10,687 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-21 20:56:10,687 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-21 20:56:10,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:56:10,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:56:10,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:10,705 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/ymvw3hhs-wandb-metadata.json |
| 2025-01-21 20:56:10,721 INFO wandb-upload_2:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/dxz3pcgn-diff.patch |
| 2025-01-21 20:56:10,737 INFO HandlerThread:2643314 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/tb" |
| save: true |
| } |
| control { |
| relay_id: "ff0654662fbf43b68afe3b9bae7d9643" |
| } |
| _info { |
| stream_id: "37t5ymwr" |
| } |
|
|
| 2025-01-21 20:56:10,737 DEBUG HandlerThread:2643314 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-21 20:56:10,746 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/ry0xbikx-code/FISH/eval_robot.py |
| 2025-01-21 20:56:10,776 DEBUG SenderThread:2643314 [sender.py:send():379] send: telemetry |
| 2025-01-21 20:56:10,788 DEBUG SenderThread:2643314 [sender.py:send():379] send: tbrecord |
| 2025-01-21 20:56:10,790 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:56:10,791 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 with policy live |
| 2025-01-21 20:56:11,009 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/z2442873-events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:56:11,258 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/diff.patch |
| 2025-01-21 20:56:11,258 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/code/FISH/eval_robot.py |
| 2025-01-21 20:56:11,258 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/requirements.txt |
| 2025-01-21 20:56:11,258 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-metadata.json |
| 2025-01-21 20:56:11,259 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:56:11,259 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:11,259 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/code/FISH |
| 2025-01-21 20:56:11,259 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/code |
| 2025-01-21 20:56:11,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:12,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:13,257 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:13,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:14,612 DEBUG SenderThread:2643314 [sender.py:send():379] send: config |
| 2025-01-21 20:56:14,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:15,621 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:15,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:16,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:17,258 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:17,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:18,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:19,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:20,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:20,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:21,259 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:21,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:22,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:23,260 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:23,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:24,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:25,687 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:56:25,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:56:25,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:25,785 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:26,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:27,261 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:27,692 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:28,692 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:29,692 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:30,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:31,262 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:31,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:31,694 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:32,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:33,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:34,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:35,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:36,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:37,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:37,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:38,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:39,696 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:40,687 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:56:40,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:56:40,729 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:41,696 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:42,696 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:43,135 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:43,276 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/config.yaml |
| 2025-01-21 20:56:43,276 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:43,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:44,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:45,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:46,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:47,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:48,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:48,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:49,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:50,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:51,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:52,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:53,697 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:54,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:56:54,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:55,693 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:56:55,693 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:56:55,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:56,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:57,280 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:57,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:58,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:56:59,281 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:56:59,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:00,233 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:00,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:01,698 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:02,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:03,167 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:57:03,167 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 with policy now |
| 2025-01-21 20:57:03,167 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:57:03,168 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:57:03,168 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:57:03,169 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:57:03,282 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:03,283 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 |
| 2025-01-21 20:57:03,283 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:57:03,283 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos |
| 2025-01-21 20:57:03,283 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media |
| 2025-01-21 20:57:03,283 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval |
| 2025-01-21 20:57:03,450 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/ljz8atw8-media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 |
| 2025-01-21 20:57:03,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:03,734 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/jjdfoqnu-media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 |
| 2025-01-21 20:57:04,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:05,284 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:05,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:05,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:06,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:07,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:08,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:09,286 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:09,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:10,354 DEBUG SystemMonitor:2643314 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-21 20:57:10,355 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:57:10,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:57:10,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:57:10,729 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:10,750 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:11,286 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:11,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:12,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:13,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:14,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:15,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:16,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:16,700 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:17,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:18,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:19,289 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:57:19,569 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/qoq3b4pq-events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:57:19,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:20,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:21,290 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:21,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:21,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:22,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:23,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:24,701 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:25,291 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:25,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:57:25,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:57:25,729 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:26,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:26,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:27,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:28,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:29,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:30,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:31,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:32,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:32,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:33,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:34,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:35,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:36,703 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:37,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:37,704 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:38,704 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:39,704 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:40,355 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:57:40,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:57:40,691 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:57:40,733 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:41,704 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:42,704 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:42,857 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:43,705 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:44,460 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:57:44,460 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 with policy now |
| 2025-01-21 20:57:44,460 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:57:44,461 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:57:44,461 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:57:44,461 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:57:44,705 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:44,732 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/5gdnijbn-media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 |
| 2025-01-21 20:57:45,305 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:57:45,305 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:45,306 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 |
| 2025-01-21 20:57:45,306 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval |
| 2025-01-21 20:57:45,603 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/9j33oegm-media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 |
| 2025-01-21 20:57:45,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:46,705 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:47,306 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:47,706 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:47,980 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:48,706 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:49,307 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:49,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:50,713 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:51,707 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:52,707 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:53,137 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:57:53,137 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:57:53,137 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:53,138 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:57:53,308 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:57:53,308 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:57:53,308 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:53,707 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:54,707 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:55,309 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:55,691 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:57:55,691 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:57:55,733 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:56,711 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:57,708 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:58,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:57:59,135 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:57:59,310 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:57:59,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:00,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:01,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:02,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:03,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:04,692 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:04,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:05,709 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:06,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:07,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:08,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:09,694 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:09,741 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:10,356 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:58:10,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:58:10,688 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:58:10,729 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:11,314 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:11,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:12,710 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:13,711 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:14,707 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:14,753 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:15,715 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:16,055 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:58:16,055 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 with policy now |
| 2025-01-21 20:58:16,059 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:58:16,059 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:58:16,060 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:58:16,060 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:58:16,307 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/8bx9sbuu-media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 |
| 2025-01-21 20:58:16,316 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:58:16,316 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 |
| 2025-01-21 20:58:16,316 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval |
| 2025-01-21 20:58:16,593 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/4exfffme-media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 |
| 2025-01-21 20:58:16,714 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:17,316 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:17,718 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:18,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:19,317 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:19,716 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:20,433 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:20,718 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:21,317 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:21,717 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:22,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:23,318 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:23,716 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:24,319 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:58:24,592 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/7s31klt1-events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:58:24,718 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:24,959 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:58:24,959 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:58:24,959 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:58:25,319 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:58:25,320 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:25,692 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:58:25,692 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:58:25,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:25,763 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:26,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:27,320 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:27,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:28,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:29,320 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:29,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:30,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:31,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:31,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:32,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:33,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:34,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:35,719 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:36,695 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:36,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:37,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:38,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:39,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:40,356 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:58:40,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:58:40,689 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:58:40,733 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:41,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:41,843 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:42,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:43,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:44,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:45,326 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:45,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:46,720 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:47,696 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:47,721 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:48,721 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:49,721 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:50,721 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:51,328 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:58:51,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:52,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:53,696 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:53,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:54,722 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:55,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:58:55,689 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:58:55,731 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:56,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:57,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:58,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:58:58,772 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:58:59,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:00,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:01,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:02,723 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:03,729 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:03,888 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:03,932 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:59:03,932 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 with policy now |
| 2025-01-21 20:59:03,935 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:59:03,935 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:03,935 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:03,953 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:04,203 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/xcloc3yc-media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 |
| 2025-01-21 20:59:04,332 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:04,332 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 |
| 2025-01-21 20:59:04,332 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval |
| 2025-01-21 20:59:04,667 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/6a9qdti2-media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 |
| 2025-01-21 20:59:04,734 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:05,332 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:05,748 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:06,736 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:07,733 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:08,735 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:09,334 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:09,617 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:09,738 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:10,357 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:59:10,688 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:59:10,689 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:59:10,735 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:11,334 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:11,739 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:12,738 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:13,745 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:14,742 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:14,751 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:15,742 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:16,741 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:17,743 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:18,747 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:19,025 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:19,026 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:19,026 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:19,337 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:19,338 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:59:19,338 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:19,747 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:20,605 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:20,747 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:21,338 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:21,749 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:22,748 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:23,339 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:23,757 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:24,757 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:25,339 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:25,689 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:59:25,689 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:59:25,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:25,797 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:26,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:27,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:28,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:29,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:30,755 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:31,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:31,756 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:32,756 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:33,756 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:34,756 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:35,757 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:36,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:36,757 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:37,757 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:38,786 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:39,344 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:39,624 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:59:39,624 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 with policy now |
| 2025-01-21 20:59:39,628 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:59:39,628 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:39,629 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:39,629 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:39,786 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:39,882 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/23e9vvo8-media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 |
| 2025-01-21 20:59:40,344 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:40,345 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 |
| 2025-01-21 20:59:40,345 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval |
| 2025-01-21 20:59:40,366 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 20:59:40,647 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/v9dsyv5q-media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 |
| 2025-01-21 20:59:40,699 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:59:40,700 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:59:40,788 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:41,345 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:41,795 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:41,808 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:42,799 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:43,793 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:44,799 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:45,346 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:45,796 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:46,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:47,061 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:47,061 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:47,061 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:47,061 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:47,086 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 20:59:47,086 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 20:59:47,238 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:59:47,347 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:47,347 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:59:47,347 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:47,347 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-21 20:59:47,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media |
| 2025-01-21 20:59:47,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table |
| 2025-01-21 20:59:47,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval |
| 2025-01-21 20:59:47,545 INFO wandb-upload_2:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/qdrrd83r-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-21 20:59:47,556 INFO wandb-upload_1:2643314 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpa2qxpetd |
| 2025-01-21 20:59:47,616 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/edae6jwi-events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:59:47,644 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 20:59:47,654 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 20:59:47,802 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:48,034 INFO SenderThread:2643314 [sender.py:send_request_log_artifact():1525] logged artifact run-37t5ymwr-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzYzNjk0MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE4OTg3NA==', 'latestArtifact': None}} |
| 2025-01-21 20:59:48,035 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:59:48,035 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json with policy now |
| 2025-01-21 20:59:48,035 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:48,035 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,035 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,036 DEBUG SenderThread:2643314 [sender.py:send():379] send: telemetry |
| 2025-01-21 20:59:48,036 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,039 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,040 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,040 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,040 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,040 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,040 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,041 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,041 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,042 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,042 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,042 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,043 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,043 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,043 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,043 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,043 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,044 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,044 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,044 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,044 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,045 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:48,045 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 20:59:48,299 INFO wandb-upload_2:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/codu6ha8-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-21 20:59:48,347 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:48,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 20:59:48,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json |
| 2025-01-21 20:59:48,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval |
| 2025-01-21 20:59:48,510 INFO wandb-upload_0:2643314 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdiqzwyf7 |
| 2025-01-21 20:59:48,528 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/vcx0yrho-media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json |
| 2025-01-21 20:59:48,802 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:48,926 INFO SenderThread:2643314 [sender.py:send_request_log_artifact():1525] logged artifact run-37t5ymwr-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzYzNjk2NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE4OTg4Mw==', 'latestArtifact': None}} |
| 2025-01-21 20:59:48,926 DEBUG SenderThread:2643314 [sender.py:send():379] send: files |
| 2025-01-21 20:59:48,926 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json with policy now |
| 2025-01-21 20:59:48,927 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 20:59:48,927 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 20:59:48,927 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 20:59:49,238 INFO wandb-upload_2:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/4o86scus-media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json |
| 2025-01-21 20:59:49,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 20:59:49,348 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 20:59:49,802 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:50,802 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:51,802 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:52,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:52,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:53,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:54,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:55,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 20:59:55,690 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 20:59:55,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:56,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:57,773 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 20:59:57,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:58,352 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/config.yaml |
| 2025-01-21 20:59:58,803 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 20:59:59,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:00,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:01,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:02,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:03,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:00:03,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:04,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:05,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:06,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:07,804 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:08,702 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:00:08,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:09,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:10,358 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 21:00:10,690 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:00:10,691 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:00:10,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:11,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:12,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:13,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:13,840 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:00:14,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:15,805 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:16,834 DEBUG SenderThread:2643314 [sender.py:send():379] send: exit |
| 2025-01-21 21:00:16,834 INFO SenderThread:2643314 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-21 21:00:16,834 INFO SenderThread:2643314 [sender.py:send_exit():588] handling runtime: 246 |
| 2025-01-21 21:00:16,834 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:00:16,834 INFO SenderThread:2643314 [sender.py:send_exit():594] send defer |
| 2025-01-21 21:00:16,834 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:16,834 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-21 21:00:16,835 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:16,835 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-21 21:00:16,835 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-21 21:00:16,835 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:16,835 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-21 21:00:16,835 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:16,835 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-21 21:00:16,835 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-21 21:00:16,835 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:16,835 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-21 21:00:16,835 INFO HandlerThread:2643314 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-21 21:00:16,835 DEBUG SystemMonitor:2643314 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-21 21:00:16,835 DEBUG SystemMonitor:2643314 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-21 21:00:16,836 INFO HandlerThread:2643314 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-21 21:00:16,836 INFO HandlerThread:2643314 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-21 21:00:16,889 INFO HandlerThread:2643314 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-21 21:00:16,890 INFO HandlerThread:2643314 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-21 21:00:16,890 INFO HandlerThread:2643314 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-21 21:00:16,890 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:16,890 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-21 21:00:16,890 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-21 21:00:16,890 DEBUG SenderThread:2643314 [sender.py:send():379] send: stats |
| 2025-01-21 21:00:16,891 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:16,891 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-21 21:00:16,891 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:16,891 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-21 21:00:16,891 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-21 21:00:16,891 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:16,891 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-21 21:00:17,359 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 21:00:19,360 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 21:00:23,067 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:23,067 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:00:23,068 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-21 21:00:23,069 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:23,069 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-21 21:00:23,069 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-21 21:00:23,069 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:23,069 DEBUG SenderThread:2643314 [sender.py:send():379] send: history |
| 2025-01-21 21:00:23,069 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:00:23,069 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:23,069 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-21 21:00:23,071 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:00:23,071 DEBUG SenderThread:2643314 [sender.py:send():379] send: summary |
| 2025-01-21 21:00:23,075 INFO SenderThread:2643314 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:00:23,075 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:23,075 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-21 21:00:23,075 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-21 21:00:23,075 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:23,075 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-21 21:00:23,075 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:23,075 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-21 21:00:23,076 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-21 21:00:23,076 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:00:23,076 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:23,076 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-21 21:00:23,076 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:23,076 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-21 21:00:23,362 INFO Thread-12 :2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 21:00:23,835 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:26,491 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-21 21:00:26,491 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:26,492 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:26,492 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-21 21:00:26,492 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:26,492 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-21 21:00:26,492 INFO SenderThread:2643314 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-21 21:00:26,494 INFO SenderThread:2643314 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-21 21:00:26,496 INFO SenderThread:2643314 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-21 21:00:26,509 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-21 21:00:26,509 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:26,509 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-21 21:00:26,509 DEBUG SenderThread:2643314 [sender.py:send():379] send: artifact |
| 2025-01-21 21:00:26,770 INFO SenderThread:2643314 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1NzYxOTQ3OQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1NzYyNTUzOQ==', 'versionIndex': 454}}} |
| 2025-01-21 21:00:26,770 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:26,770 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-21 21:00:26,770 INFO SenderThread:2643314 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-21 21:00:26,836 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:27,363 INFO SenderThread:2643314 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 21:00:27,363 INFO SenderThread:2643314 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files |
| 2025-01-21 21:00:27,363 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/config.yaml config.yaml |
| 2025-01-21 21:00:27,363 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 21:00:27,364 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json wandb-summary.json |
| 2025-01-21 21:00:27,364 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/requirements.txt requirements.txt |
| 2025-01-21 21:00:27,364 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-21 21:00:27,364 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log output.log |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/diff.patch diff.patch |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 media/videos/eval/1_eval_1_52f38aea11256dff566a.mp4 |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 media/videos/eval/2_eval_2_02b7b0943e5ffceb2ddf.mp4 |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 media/videos/eval/4_eval_4_e88bcd3f25d83f27b5d9.mp4 |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 media/videos/eval/0_eval_0_2f683e3d27762b787fec.mp4 |
| 2025-01-21 21:00:27,366 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 media/videos/eval/3_eval_3_eb1f7174555f6baf68df.mp4 |
| 2025-01-21 21:00:27,367 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json media/table/eval/total_success_rate_ci_6_0c9cd2f25aeded733b20.table.json |
| 2025-01-21 21:00:27,367 INFO SenderThread:2643314 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-21 21:00:27,367 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-21 21:00:27,367 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:27,367 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:27,367 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-21 21:00:27,373 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:27,373 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-21 21:00:27,373 INFO SenderThread:2643314 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:00:27,571 INFO wandb-upload_1:2643314 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/config.yaml |
| 2025-01-21 21:00:27,607 INFO wandb-upload_0:2643314 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/wandb-summary.json |
| 2025-01-21 21:00:27,614 INFO wandb-upload_4:2643314 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/output.log |
| 2025-01-21 21:00:27,614 INFO wandb-upload_3:2643314 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/files/requirements.txt |
| 2025-01-21 21:00:27,651 INFO wandb-upload_2:2643314 [upload_job.py:push():130] Uploaded file /tmp/tmphymvqtsywandb/l08znhm2-events.out.tfevents.1737510970.leonmkim-ROG-Strix-G15CS-G15CS.2643122.0 |
| 2025-01-21 21:00:27,836 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:27,836 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:27,851 INFO Thread-11 (_thread_body):2643314 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-21 21:00:27,851 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:27,851 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-21 21:00:27,851 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:27,852 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-21 21:00:27,852 INFO SenderThread:2643314 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-21 21:00:27,853 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-21 21:00:27,853 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:27,853 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-21 21:00:27,853 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:27,853 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-21 21:00:27,853 INFO SenderThread:2643314 [file_stream.py:finish():601] file stream finish called |
| 2025-01-21 21:00:28,051 INFO SenderThread:2643314 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-21 21:00:28,051 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-21 21:00:28,051 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:28,051 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-21 21:00:28,051 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:28,051 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-21 21:00:28,051 INFO SenderThread:2643314 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-21 21:00:28,051 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:00:28,051 INFO HandlerThread:2643314 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-21 21:00:28,052 DEBUG SenderThread:2643314 [sender.py:send():379] send: final |
| 2025-01-21 21:00:28,052 DEBUG SenderThread:2643314 [sender.py:send():379] send: footer |
| 2025-01-21 21:00:28,052 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:00:28,052 INFO SenderThread:2643314 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-21 21:00:28,052 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:28,052 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:28,053 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-21 21:00:28,053 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: server_info |
| 2025-01-21 21:00:28,055 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:00:28,055 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-21 21:00:28,055 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-21 21:00:28,056 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:00:28,100 DEBUG SenderThread:2643314 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:00:28,100 INFO MainThread:2643314 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-21 21:00:28,101 INFO MainThread:2643314 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-21 21:00:28,101 INFO MainThread:2643314 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-21 21:00:28,101 DEBUG HandlerThread:2643314 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-21 21:00:28,101 INFO HandlerThread:2643314 [handler.py:finish():882] shutting down handler |
| 2025-01-21 21:00:29,055 INFO WriterThread:2643314 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_0/37t5ymwr/wandb/run-20250121_205609-37t5ymwr/run-37t5ymwr.wandb |
| 2025-01-21 21:00:29,100 INFO SenderThread:2643314 [sender.py:finish():1615] shutting down sender |
| 2025-01-21 21:00:29,100 INFO SenderThread:2643314 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:00:29,100 INFO SenderThread:2643314 [file_pusher.py:join():175] waiting for file pusher |
|
|
