| 2025-01-01 10:51:43,767 INFO StreamThr :1786163 [internal.py:wandb_internal():85] W&B internal server running at pid: 1786163, started at: 2025-01-01 10:51:43.766599 |
| 2025-01-01 10:51:43,768 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status |
| 2025-01-01 10:51:43,779 INFO WriterThread:1786163 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/run-qo464509.wandb |
| 2025-01-01 10:51:43,780 DEBUG SenderThread:1786163 [sender.py:send():379] send: header |
| 2025-01-01 10:51:43,787 DEBUG SenderThread:1786163 [sender.py:send():379] send: run |
| 2025-01-01 10:51:44,066 INFO SenderThread:1786163 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files |
| 2025-01-01 10:51:44,066 INFO SenderThread:1786163 [sender.py:_start_run_threads():1188] run started: qo464509 with start time 1735746703.767215 |
| 2025-01-01 10:51:44,083 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-01 10:51:44,084 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: check_version |
| 2025-01-01 10:51:44,159 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-01 10:51:44,164 DEBUG HandlerThread:1786163 [system_info.py:__init__():26] System info init |
| 2025-01-01 10:51:44,164 DEBUG HandlerThread:1786163 [system_info.py:__init__():41] System info init done |
| 2025-01-01 10:51:44,164 INFO HandlerThread:1786163 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-01 10:51:44,164 INFO SystemMonitor:1786163 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-01 10:51:44,164 INFO HandlerThread:1786163 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-01 10:51:44,165 INFO SystemMonitor:1786163 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-01 10:51:44,166 INFO SystemMonitor:1786163 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-01 10:51:44,166 INFO SystemMonitor:1786163 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-01 10:51:44,167 INFO SystemMonitor:1786163 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-01 10:51:44,167 INFO SystemMonitor:1786163 [interfaces.py:start():188] Started network monitoring |
| 2025-01-01 10:51:44,177 DEBUG HandlerThread:1786163 [system_info.py:probe():152] Probing system |
| 2025-01-01 10:51:44,179 DEBUG HandlerThread:1786163 [system_info.py:_probe_git():137] Probing git |
| 2025-01-01 10:51:44,183 DEBUG HandlerThread:1786163 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-01 10:51:44,183 DEBUG HandlerThread:1786163 [system_info.py:probe():200] Probing system done |
| 2025-01-01 10:51:44,183 DEBUG HandlerThread:1786163 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T15:51:44.177372', 'startedAt': '2025-01-01T15:51:43.758070', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.66475, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.988, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4497.309, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 650.9016075134277}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-01 10:51:44,183 INFO HandlerThread:1786163 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-01 10:51:44,184 INFO HandlerThread:1786163 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-01 10:51:44,184 DEBUG HandlerThread:1786163 [system_info.py:_save_code():44] Saving code |
| 2025-01-01 10:51:44,185 DEBUG HandlerThread:1786163 [system_info.py:_save_code():67] Saving code done |
| 2025-01-01 10:51:44,185 DEBUG HandlerThread:1786163 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-01 10:51:44,255 DEBUG HandlerThread:1786163 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-01 10:51:44,255 INFO HandlerThread:1786163 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-01 10:51:44,259 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:51:44,259 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-01 10:51:44,259 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-01 10:51:44,259 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-01 10:51:44,534 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/3ixjw6p7-diff.patch |
| 2025-01-01 10:51:44,538 INFO wandb-upload_0:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/sp2l1kdn-wandb-metadata.json |
| 2025-01-01 10:51:44,542 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/m0ho2arv-code/FISH/eval_robot.py |
| 2025-01-01 10:51:44,547 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-01 10:51:44,547 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-01 10:51:44,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:44,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:51:44,549 DEBUG SenderThread:1786163 [sender.py:send():379] send: telemetry |
| 2025-01-01 10:51:44,549 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:51:44,597 INFO HandlerThread:1786163 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/tb" |
| save: true |
| } |
| control { |
| relay_id: "3d2e6f45d36949e79d82444785edfe6c" |
| } |
| _info { |
| stream_id: "qo464509" |
| } |
|
|
| 2025-01-01 10:51:44,597 DEBUG HandlerThread:1786163 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-01 10:51:44,676 DEBUG SenderThread:1786163 [sender.py:send():379] send: tbrecord |
| 2025-01-01 10:51:44,677 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:51:44,677 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 with policy live |
| 2025-01-01 10:51:44,861 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/rng1gizz-events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/diff.patch |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/code/FISH/eval_robot.py |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/requirements.txt |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-metadata.json |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/code |
| 2025-01-01 10:51:45,068 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/code/FISH |
| 2025-01-01 10:51:45,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:46,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:47,067 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:51:47,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:48,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:49,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:51:49,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:50,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:51,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:52,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:53,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:54,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:51:54,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:55,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:56,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:57,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:58,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:59,547 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:51:59,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:51:59,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:51:59,675 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:00,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:01,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:02,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:03,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:04,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:05,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:05,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:06,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:07,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:08,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:09,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:10,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:10,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:11,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:12,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:13,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:14,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:52:14,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:52:14,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:15,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:15,610 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:16,074 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/config.yaml |
| 2025-01-01 10:52:16,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:17,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:18,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:19,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:20,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:21,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:21,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:22,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:23,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:24,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:25,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:26,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:27,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:27,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:28,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:29,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:52:29,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:52:29,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:30,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:31,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:32,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:32,685 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:33,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:34,331 DEBUG SenderThread:1786163 [sender.py:send():379] send: config |
| 2025-01-01 10:52:34,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:35,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:36,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:37,080 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:37,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:38,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:38,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:39,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:40,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:41,081 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:41,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:42,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:43,082 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:43,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:43,755 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:44,168 DEBUG SystemMonitor:1786163 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-01 10:52:44,169 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:52:44,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:52:44,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:52:44,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:45,082 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:45,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:46,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:47,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:48,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:48,768 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:49,083 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/config.yaml |
| 2025-01-01 10:52:49,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:50,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:51,084 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:51,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:52,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:53,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:54,522 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:52:54,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:55,085 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:52:55,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:56,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:57,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:58,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:59,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:52:59,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:52:59,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:52:59,687 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:00,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:01,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:02,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:03,087 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:03,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:04,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:05,088 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:05,089 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:05,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:06,557 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:07,089 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:07,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:08,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:09,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:10,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:10,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:11,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:12,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:13,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:14,169 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:53:14,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:53:14,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:53:14,593 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:15,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:15,614 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:16,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:17,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:18,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:19,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:20,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:20,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:21,093 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:21,559 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:22,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:23,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:24,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:25,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:26,552 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:26,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:27,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:28,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:29,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:53:29,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:53:29,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:30,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:31,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:31,679 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:32,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:33,096 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:33,158 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:53:33,159 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 with policy now |
| 2025-01-01 10:53:33,159 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:53:33,160 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:53:33,160 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:53:33,162 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:53:33,387 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/2mla3ytc-media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 |
| 2025-01-01 10:53:33,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:34,096 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 |
| 2025-01-01 10:53:34,097 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:53:34,097 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval |
| 2025-01-01 10:53:34,097 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media |
| 2025-01-01 10:53:34,097 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos |
| 2025-01-01 10:53:34,401 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/dg1j6msb-media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 |
| 2025-01-01 10:53:34,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:35,097 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:35,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:36,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:37,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:37,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:38,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:39,098 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:39,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:40,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:41,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:42,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:43,100 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:43,100 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:53:43,116 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:43,351 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/32fstrdh-events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:53:43,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:44,169 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:53:44,548 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:53:44,548 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:53:44,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:45,100 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:45,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:46,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:47,101 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:53:47,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:48,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:49,071 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:49,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:50,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:51,564 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:52,564 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:53,564 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:54,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:53:54,564 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:55,565 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:56,565 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:57,565 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:58,565 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:59,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:53:59,550 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:53:59,593 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:53:59,610 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:00,572 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:01,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:02,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:03,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:04,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:05,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:05,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:06,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:07,568 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:08,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:09,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:10,553 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:10,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:11,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:12,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:13,569 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:14,170 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:54:14,288 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:54:14,288 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 with policy now |
| 2025-01-01 10:54:14,289 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:54:14,290 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:54:14,290 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:54:14,290 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:54:14,514 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/zf34tc2c-media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 |
| 2025-01-01 10:54:14,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:54:14,549 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:54:14,570 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:15,110 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:15,110 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:54:15,110 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 |
| 2025-01-01 10:54:15,110 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval |
| 2025-01-01 10:54:15,378 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/tao80fiw-media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 |
| 2025-01-01 10:54:15,574 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:15,627 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:16,110 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:16,577 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:17,577 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:18,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:19,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:20,576 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:21,112 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:21,565 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:21,587 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:22,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:23,112 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:23,113 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:54:23,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:24,099 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:54:24,099 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:54:24,099 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:54:24,113 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:54:24,577 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:25,113 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:25,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:26,577 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:26,623 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:27,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:28,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:29,115 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:29,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:54:29,549 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:54:29,593 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:30,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:31,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:31,700 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:32,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:33,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:34,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:35,578 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:36,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:37,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:37,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:38,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:39,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:40,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:41,579 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:42,554 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:42,580 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:43,580 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:44,170 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:54:44,549 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:54:44,549 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:54:44,593 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:45,580 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:46,580 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:47,580 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:47,636 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:48,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:49,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:50,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:51,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:52,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:53,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:53,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:54,582 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:54,925 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:54:54,926 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:54:54,926 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 with policy now |
| 2025-01-01 10:54:54,926 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:54:54,926 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:54:54,926 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:54:55,123 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:55,123 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:54:55,123 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 |
| 2025-01-01 10:54:55,123 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval |
| 2025-01-01 10:54:55,174 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/8aa7ymy1-media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 |
| 2025-01-01 10:54:55,417 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/gtz2io06-media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 |
| 2025-01-01 10:54:55,585 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:56,123 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:56,591 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:57,124 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:54:57,589 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:58,590 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:54:58,960 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:54:59,555 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:54:59,555 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:54:59,601 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:00,598 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:01,125 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:01,602 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:02,590 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:03,592 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:04,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:04,598 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:05,126 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:05,598 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:06,593 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:07,595 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:08,599 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:09,600 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:10,560 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:10,597 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:11,597 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:12,598 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:13,601 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:14,171 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:55:14,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:55:14,551 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:55:14,598 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:14,922 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:55:14,922 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:55:14,922 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:55:15,130 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:55:15,130 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:15,130 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:55:15,364 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/2zbmys5t-events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:55:15,602 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:16,249 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:16,604 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:17,130 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:17,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:18,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:19,131 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:19,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:20,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:21,132 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:21,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:21,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:22,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:23,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:24,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:25,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:26,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:26,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:27,605 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:28,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:29,550 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:55:29,551 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:55:29,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:30,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:31,135 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:31,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:31,681 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:32,608 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:33,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:34,606 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:35,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:36,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:37,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:37,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:38,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:39,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:40,607 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:41,609 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:42,558 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:42,608 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:43,608 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:44,171 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:55:44,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:55:44,551 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:55:44,608 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:45,609 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:46,337 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:55:46,337 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 with policy now |
| 2025-01-01 10:55:46,342 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:55:46,343 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:55:46,343 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:55:46,343 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:55:46,547 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/ozvfccbj-media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 |
| 2025-01-01 10:55:46,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:47,141 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:47,141 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:55:47,141 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 |
| 2025-01-01 10:55:47,141 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval |
| 2025-01-01 10:55:47,403 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/q58ldh30-media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 |
| 2025-01-01 10:55:47,622 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:48,375 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:48,613 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:49,614 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:50,616 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:51,615 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:52,630 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:53,143 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:53,615 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:53,635 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:54,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:54,970 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:55:54,970 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:55:54,970 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:55:55,143 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:55:55,144 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:55,144 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:55:55,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:56,623 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:57,144 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:55:57,618 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:58,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:55:58,669 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:55:59,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:55:59,551 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:55:59,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:00,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:01,146 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:01,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:02,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:03,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:04,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:04,619 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:05,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:06,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:07,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:08,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:09,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:09,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:10,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:11,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:12,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:13,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:14,171 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:56:14,551 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:56:14,551 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:56:14,620 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:14,679 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:15,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:16,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:17,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:18,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:19,152 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:19,621 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:20,561 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:20,622 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:21,622 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:22,622 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:23,622 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:24,623 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:25,562 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:25,623 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:26,630 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:26,990 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:56:26,990 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 with policy now |
| 2025-01-01 10:56:26,995 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:56:26,996 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:56:26,996 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:26,996 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:27,154 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:27,154 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:27,155 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 |
| 2025-01-01 10:56:27,155 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval |
| 2025-01-01 10:56:27,210 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/w1vs6v9d-media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 |
| 2025-01-01 10:56:27,508 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/yclzgvqr-media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 |
| 2025-01-01 10:56:27,645 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:28,630 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:29,155 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:29,563 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:56:29,563 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:56:29,631 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:30,637 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:30,646 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:31,635 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:32,634 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:33,156 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:33,670 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:34,657 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:35,650 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:35,710 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:36,643 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:37,158 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:37,648 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:38,663 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:39,649 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:40,656 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:41,581 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:41,646 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:42,652 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:43,656 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:44,172 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:56:44,556 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 10:56:44,556 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 10:56:44,654 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:45,654 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:46,648 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:46,664 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:47,657 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:48,661 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:49,683 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:50,035 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:56:50,035 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:50,035 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:50,162 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:56:50,162 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:50,246 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 10:56:50,247 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 10:56:50,388 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:56:50,399 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/3qtsnd0z-events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:56:50,659 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:50,671 INFO wandb-upload_2:1786163 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1r6scxd5 |
| 2025-01-01 10:56:51,022 INFO SenderThread:1786163 [sender.py:send_request_log_artifact():1525] logged artifact run-qo464509-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY2Njg5Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTIzOTEyMQ==', 'latestArtifact': None}} |
| 2025-01-01 10:56:51,022 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:56:51,022 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-01 10:56:51,023 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:56:51,023 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:51,023 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:51,162 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:51,163 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:51,163 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 10:56:51,163 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media |
| 2025-01-01 10:56:51,163 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table |
| 2025-01-01 10:56:51,163 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval |
| 2025-01-01 10:56:51,227 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/xj491i9b-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 10:56:51,488 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/r93183fk-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 10:56:51,660 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:52,024 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:56:52,660 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:53,428 DEBUG SenderThread:1786163 [sender.py:send():379] send: telemetry |
| 2025-01-01 10:56:53,428 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,429 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,430 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,430 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,430 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,431 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,435 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,435 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,436 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,436 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,441 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,441 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,441 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,442 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,448 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,449 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,449 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,450 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,456 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,456 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,457 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:53,458 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:53,537 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 10:56:53,538 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 10:56:53,547 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 10:56:53,660 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:53,889 INFO wandb-upload_1:1786163 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpwb2orbjw |
| 2025-01-01 10:56:54,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:56:54,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:54,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json |
| 2025-01-01 10:56:54,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval |
| 2025-01-01 10:56:54,287 INFO SenderThread:1786163 [sender.py:send_request_log_artifact():1525] logged artifact run-qo464509-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY2NjkzOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTIzOTEyNw==', 'latestArtifact': None}} |
| 2025-01-01 10:56:54,287 DEBUG SenderThread:1786163 [sender.py:send():379] send: files |
| 2025-01-01 10:56:54,287 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json with policy now |
| 2025-01-01 10:56:54,288 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:56:54,289 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:56:54,290 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:54,461 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/et2enguc-media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json |
| 2025-01-01 10:56:54,660 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:56:54,695 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/93m5r6pj-media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json |
| 2025-01-01 10:56:55,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:56:55,164 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:55,361 DEBUG SenderThread:1786163 [sender.py:send():379] send: exit |
| 2025-01-01 10:56:55,361 INFO SenderThread:1786163 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-01 10:56:55,362 INFO SenderThread:1786163 [sender.py:send_exit():588] handling runtime: 311 |
| 2025-01-01 10:56:55,362 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:56:55,362 INFO SenderThread:1786163 [sender.py:send_exit():594] send defer |
| 2025-01-01 10:56:55,362 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:56:55,362 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-01 10:56:55,362 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:56:55,362 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-01 10:56:55,362 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-01 10:56:55,363 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:56:55,363 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-01 10:56:55,363 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:56:55,363 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-01 10:56:55,363 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-01 10:56:55,363 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:56:55,363 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-01 10:56:55,363 INFO HandlerThread:1786163 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-01 10:56:55,363 DEBUG SystemMonitor:1786163 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-01 10:56:55,363 INFO HandlerThread:1786163 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-01 10:56:55,364 DEBUG SystemMonitor:1786163 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-01 10:56:55,364 INFO HandlerThread:1786163 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-01 10:56:55,415 INFO HandlerThread:1786163 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-01 10:56:55,415 INFO HandlerThread:1786163 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-01 10:56:55,415 INFO HandlerThread:1786163 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-01 10:56:55,416 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:56:55,416 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-01 10:56:55,416 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-01 10:56:55,416 DEBUG SenderThread:1786163 [sender.py:send():379] send: stats |
| 2025-01-01 10:56:55,416 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:56:55,416 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-01 10:56:55,416 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:56:55,417 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-01 10:56:55,417 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-01 10:56:55,417 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:56:55,417 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-01 10:56:56,165 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:56:57,165 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:57:02,044 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:02,044 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:57:02,045 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-01 10:57:02,046 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:02,046 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-01 10:57:02,046 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-01 10:57:02,046 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:02,046 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:02,046 DEBUG SenderThread:1786163 [sender.py:send():379] send: history |
| 2025-01-01 10:57:02,046 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-01 10:57:02,046 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 10:57:02,047 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:57:02,048 DEBUG SenderThread:1786163 [sender.py:send():379] send: summary |
| 2025-01-01 10:57:02,051 INFO SenderThread:1786163 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 10:57:02,051 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:02,052 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-01 10:57:02,052 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-01 10:57:02,052 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:02,052 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-01 10:57:02,052 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:02,052 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-01 10:57:02,055 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 10:57:02,166 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-01 10:57:02,166 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:02,166 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-01 10:57:02,167 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:02,167 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-01 10:57:02,167 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/config.yaml |
| 2025-01-01 10:57:02,167 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:57:02,362 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:04,074 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-01 10:57:04,074 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:04,074 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:04,075 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-01 10:57:04,075 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:04,075 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-01 10:57:04,075 INFO SenderThread:1786163 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-01 10:57:04,077 INFO SenderThread:1786163 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-01 10:57:04,079 INFO SenderThread:1786163 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-01 10:57:04,093 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-01 10:57:04,093 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:04,093 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-01 10:57:04,093 DEBUG SenderThread:1786163 [sender.py:send():379] send: artifact |
| 2025-01-01 10:57:04,168 INFO Thread-12 :1786163 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:57:04,363 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:04,632 INFO wandb-upload_1:1786163 [upload_job.py:push():85] Skipped uploading /tmp/tmpqfdli0iz/wandb-job.json |
| 2025-01-01 10:57:04,633 INFO wandb-upload_0:1786163 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmprokw9jhr |
| 2025-01-01 10:57:04,768 INFO wandb-upload_2:1786163 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp8m6d6qiy |
| 2025-01-01 10:57:05,264 INFO SenderThread:1786163 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDIyMjk2Ng==', 'versionIndex': 403}}} |
| 2025-01-01 10:57:05,264 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:05,264 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-01 10:57:05,264 INFO SenderThread:1786163 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-01 10:57:06,169 INFO SenderThread:1786163 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files |
| 2025-01-01 10:57:06,169 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/config.yaml config.yaml |
| 2025-01-01 10:57:06,169 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json wandb-summary.json |
| 2025-01-01 10:57:06,169 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:57:06,171 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/requirements.txt requirements.txt |
| 2025-01-01 10:57:06,171 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-01 10:57:06,171 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log output.log |
| 2025-01-01 10:57:06,171 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/diff.patch diff.patch |
| 2025-01-01 10:57:06,175 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 media/videos/eval/1_eval_1_561430e6869e3e143edf.mp4 |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 media/videos/eval/3_eval_3_a5ed70decb9e07902125.mp4 |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 media/videos/eval/4_eval_4_0f9d89dde9cf486c83fd.mp4 |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 media/videos/eval/0_eval_0_047a5015a7283e01ecf8.mp4 |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 media/videos/eval/2_eval_2_89284451a45c1f0bdd58.mp4 |
| 2025-01-01 10:57:06,180 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json media/table/eval/total_success_rate_ci_6_1a02426d42619cd057a5.table.json |
| 2025-01-01 10:57:06,181 INFO SenderThread:1786163 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 10:57:06,181 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-01 10:57:06,181 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:06,181 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:06,181 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-01 10:57:06,181 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:06,181 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-01 10:57:06,181 INFO SenderThread:1786163 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 10:57:06,344 INFO wandb-upload_1:1786163 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/config.yaml |
| 2025-01-01 10:57:06,360 INFO wandb-upload_2:1786163 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/wandb-summary.json |
| 2025-01-01 10:57:06,364 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:06,364 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:06,401 INFO wandb-upload_3:1786163 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/requirements.txt |
| 2025-01-01 10:57:06,410 INFO wandb-upload_4:1786163 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/files/output.log |
| 2025-01-01 10:57:06,443 INFO wandb-upload_0:1786163 [upload_job.py:push():130] Uploaded file /tmp/tmphoej7q7fwandb/uxvw1l74-events.out.tfevents.1735746704.leonmkim-ROG-Strix-G15CS-G15CS.1785977.0 |
| 2025-01-01 10:57:06,644 INFO Thread-11 (_thread_body):1786163 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-01 10:57:06,644 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:06,644 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-01 10:57:06,644 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:06,644 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-01 10:57:06,644 INFO SenderThread:1786163 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-01 10:57:06,645 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-01 10:57:06,645 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:06,645 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-01 10:57:06,646 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:06,646 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-01 10:57:06,646 INFO SenderThread:1786163 [file_stream.py:finish():601] file stream finish called |
| 2025-01-01 10:57:06,839 INFO SenderThread:1786163 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-01 10:57:06,839 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-01 10:57:06,839 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:06,839 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-01 10:57:06,839 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:06,839 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-01 10:57:06,839 INFO SenderThread:1786163 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-01 10:57:06,840 DEBUG SenderThread:1786163 [sender.py:send():379] send: final |
| 2025-01-01 10:57:06,840 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 10:57:06,840 DEBUG SenderThread:1786163 [sender.py:send():379] send: footer |
| 2025-01-01 10:57:06,840 INFO HandlerThread:1786163 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-01 10:57:06,840 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 10:57:06,840 INFO SenderThread:1786163 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-01 10:57:06,840 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:06,841 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-01 10:57:06,841 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 10:57:06,841 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:06,842 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: server_info |
| 2025-01-01 10:57:06,843 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-01 10:57:06,844 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-01 10:57:06,845 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 10:57:06,896 DEBUG SenderThread:1786163 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 10:57:06,896 INFO MainThread:1786163 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-01 10:57:06,897 INFO MainThread:1786163 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-01 10:57:06,897 INFO MainThread:1786163 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-01 10:57:06,897 DEBUG HandlerThread:1786163 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-01 10:57:06,897 INFO HandlerThread:1786163 [handler.py:finish():882] shutting down handler |
| 2025-01-01 10:57:07,841 INFO WriterThread:1786163 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/qo464509/wandb/run-20250101_105143-qo464509/run-qo464509.wandb |
| 2025-01-01 10:57:07,896 INFO SenderThread:1786163 [sender.py:finish():1615] shutting down sender |
| 2025-01-01 10:57:07,897 INFO SenderThread:1786163 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 10:57:07,897 INFO SenderThread:1786163 [file_pusher.py:join():175] waiting for file pusher |
|
|
