| 2025-01-01 11:46:17,424 INFO StreamThr :1797655 [internal.py:wandb_internal():85] W&B internal server running at pid: 1797655, started at: 2025-01-01 11:46:17.423856 |
| 2025-01-01 11:46:17,425 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status |
| 2025-01-01 11:46:17,437 INFO WriterThread:1797655 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/run-8511ppcj.wandb |
| 2025-01-01 11:46:17,437 DEBUG SenderThread:1797655 [sender.py:send():379] send: header |
| 2025-01-01 11:46:17,443 DEBUG SenderThread:1797655 [sender.py:send():379] send: run |
| 2025-01-01 11:46:17,681 INFO SenderThread:1797655 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files |
| 2025-01-01 11:46:17,681 INFO SenderThread:1797655 [sender.py:_start_run_threads():1188] run started: 8511ppcj with start time 1735749977.424992 |
| 2025-01-01 11:46:17,698 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-01 11:46:17,698 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: check_version |
| 2025-01-01 11:46:17,772 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-01 11:46:17,776 DEBUG HandlerThread:1797655 [system_info.py:__init__():26] System info init |
| 2025-01-01 11:46:17,776 DEBUG HandlerThread:1797655 [system_info.py:__init__():41] System info init done |
| 2025-01-01 11:46:17,776 INFO HandlerThread:1797655 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-01 11:46:17,776 INFO SystemMonitor:1797655 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-01 11:46:17,777 INFO HandlerThread:1797655 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-01 11:46:17,777 INFO SystemMonitor:1797655 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-01 11:46:17,777 INFO SystemMonitor:1797655 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-01 11:46:17,778 INFO SystemMonitor:1797655 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-01 11:46:17,779 INFO SystemMonitor:1797655 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-01 11:46:17,779 INFO SystemMonitor:1797655 [interfaces.py:start():188] Started network monitoring |
| 2025-01-01 11:46:17,789 DEBUG HandlerThread:1797655 [system_info.py:probe():152] Probing system |
| 2025-01-01 11:46:17,790 DEBUG HandlerThread:1797655 [system_info.py:_probe_git():137] Probing git |
| 2025-01-01 11:46:17,795 DEBUG HandlerThread:1797655 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-01 11:46:17,795 DEBUG HandlerThread:1797655 [system_info.py:probe():200] Probing system done |
| 2025-01-01 11:46:17,795 DEBUG HandlerThread:1797655 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T16:46:17.789371', 'startedAt': '2025-01-01T16:46:17.415416', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3751.5314999999996, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4505.388, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 724.0196418762207}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-01 11:46:17,795 INFO HandlerThread:1797655 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-01 11:46:17,795 INFO HandlerThread:1797655 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-01 11:46:17,795 DEBUG HandlerThread:1797655 [system_info.py:_save_code():44] Saving code |
| 2025-01-01 11:46:17,796 DEBUG HandlerThread:1797655 [system_info.py:_save_code():67] Saving code done |
| 2025-01-01 11:46:17,796 DEBUG HandlerThread:1797655 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-01 11:46:17,858 DEBUG HandlerThread:1797655 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-01 11:46:17,858 INFO HandlerThread:1797655 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-01 11:46:17,862 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:46:17,862 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-01 11:46:17,862 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-01 11:46:17,862 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-01 11:46:18,094 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-01 11:46:18,094 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:46:18,094 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-01 11:46:18,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:46:18,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:18,097 INFO HandlerThread:1797655 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/tb" |
| save: true |
| } |
| control { |
| relay_id: "19a8f7b8037f4a2fb815168a54265f74" |
| } |
| _info { |
| stream_id: "8511ppcj" |
| } |
|
|
| 2025-01-01 11:46:18,097 DEBUG HandlerThread:1797655 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-01 11:46:18,123 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/7yjhb4lx-wandb-metadata.json |
| 2025-01-01 11:46:18,157 DEBUG SenderThread:1797655 [sender.py:send():379] send: telemetry |
| 2025-01-01 11:46:18,157 DEBUG SenderThread:1797655 [sender.py:send():379] send: tbrecord |
| 2025-01-01 11:46:18,158 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/i7bjapkn-diff.patch |
| 2025-01-01 11:46:18,158 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:46:18,158 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 with policy live |
| 2025-01-01 11:46:18,189 INFO wandb-upload_1:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/lygq3cfw-code/FISH/eval_robot.py |
| 2025-01-01 11:46:18,342 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/uxgn6aox-events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:46:18,683 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:46:18,683 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/code/FISH/eval_robot.py |
| 2025-01-01 11:46:18,683 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:18,683 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/diff.patch |
| 2025-01-01 11:46:18,683 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/requirements.txt |
| 2025-01-01 11:46:18,684 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-metadata.json |
| 2025-01-01 11:46:18,684 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/code |
| 2025-01-01 11:46:18,684 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/code/FISH |
| 2025-01-01 11:46:19,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:20,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:20,682 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:21,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:22,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:23,057 DEBUG SenderThread:1797655 [sender.py:send():379] send: config |
| 2025-01-01 11:46:23,057 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:23,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:24,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:24,684 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:25,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:26,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:26,684 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:27,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:28,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:28,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:28,685 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:29,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:30,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:30,685 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:31,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:32,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:32,686 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:33,094 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:46:33,094 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:46:33,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:33,181 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:34,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:34,687 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:35,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:36,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:36,687 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:37,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:38,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:38,688 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:39,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:39,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:40,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:41,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:42,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:43,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:44,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:44,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:45,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:46,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:47,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:48,094 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:46:48,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:46:48,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:49,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:49,226 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:49,691 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/config.yaml |
| 2025-01-01 11:46:50,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:51,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:52,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:53,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:54,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:55,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:46:55,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:56,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:56,693 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:46:57,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:58,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:46:59,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:00,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:00,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:01,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:02,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:03,094 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:47:03,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:47:03,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:04,101 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:04,383 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:47:04,383 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_28199d60beae23009631.mp4 with policy now |
| 2025-01-01 11:47:04,384 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:47:04,384 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:47:04,385 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:47:04,385 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:47:04,641 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/bsau7ev0-media/videos/eval/0_eval_0_28199d60beae23009631.mp4 |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/0_eval_0_28199d60beae23009631.mp4 |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos |
| 2025-01-01 11:47:04,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval |
| 2025-01-01 11:47:04,953 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/wu0d40av-media/videos/eval/0_eval_0_28199d60beae23009631.mp4 |
| 2025-01-01 11:47:05,101 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:05,394 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:06,101 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:06,695 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:07,101 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:08,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:09,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:10,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:10,437 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:10,697 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:11,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:12,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:12,698 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:13,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:14,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:15,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:16,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:16,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:17,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:17,780 DEBUG SystemMonitor:1797655 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-01 11:47:17,781 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:47:18,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:47:18,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:47:18,141 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:19,103 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:20,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:21,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:21,216 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:22,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:23,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:24,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:25,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:26,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:27,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:27,104 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:28,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:29,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:30,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:31,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:32,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:32,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:33,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:47:33,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:47:33,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:34,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:35,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:35,705 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:47:35,944 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/ssnj77ue-events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:47:36,105 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:36,705 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:37,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:37,449 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:38,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:38,706 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:39,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:40,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:40,707 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:47:41,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:42,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:43,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:43,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:44,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:45,106 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:46,107 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:47,107 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:47,781 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:47:48,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:47:48,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:47:48,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:48,245 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:49,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:50,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:51,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:52,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:53,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:54,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:54,109 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:55,109 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:56,109 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:57,109 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:58,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:47:59,098 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:47:59,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:00,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:01,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:02,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:03,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:48:03,095 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:48:03,141 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:04,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:04,202 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:05,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:06,113 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:07,113 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:07,444 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:48:07,444 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 with policy now |
| 2025-01-01 11:48:07,445 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:48:07,445 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:48:07,445 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:48:07,446 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:48:07,705 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/hpl97859-media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 |
| 2025-01-01 11:48:07,715 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:48:07,715 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 |
| 2025-01-01 11:48:07,716 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval |
| 2025-01-01 11:48:07,999 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/27flny6g-media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 |
| 2025-01-01 11:48:08,113 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:08,716 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:09,113 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:09,716 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:10,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:10,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:11,116 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:12,113 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:12,717 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:13,114 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:14,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:15,119 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:15,409 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:16,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:16,718 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:17,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:17,782 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:48:18,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:48:18,096 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:48:18,141 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:19,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:20,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:21,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:21,157 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:22,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:23,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:23,525 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:48:23,526 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:48:23,527 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:48:23,721 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:48:23,721 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:48:24,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:24,721 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:25,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:26,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:26,483 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:26,722 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:27,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:28,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:28,722 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:29,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:30,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:31,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:32,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:32,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:33,095 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:48:33,096 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:48:33,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:34,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:35,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:36,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:37,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:37,223 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:38,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:39,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:40,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:41,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:42,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:43,100 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:43,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:44,123 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:45,124 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:46,124 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:47,124 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:47,782 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:48:48,096 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:48:48,096 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:48:48,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:48,186 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:49,125 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:50,125 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:51,125 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:52,125 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:52,730 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:53,125 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:53,424 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:54,128 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:54,659 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:48:54,659 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 with policy now |
| 2025-01-01 11:48:54,660 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:48:54,660 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:48:54,660 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:48:54,661 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:48:54,731 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:48:54,731 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:54,731 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 |
| 2025-01-01 11:48:54,731 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval |
| 2025-01-01 11:48:54,922 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/2s1mhs27-media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 |
| 2025-01-01 11:48:55,133 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:55,181 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/e1ielie5-media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 |
| 2025-01-01 11:48:56,132 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:56,731 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:48:57,133 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:58,144 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:48:58,705 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:48:59,133 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:00,135 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:00,733 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:01,140 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:02,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:02,733 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:03,102 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:49:03,102 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:49:03,145 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:04,134 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:04,171 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:05,133 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:06,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:07,139 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:08,143 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:09,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:10,107 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:10,139 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:11,141 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:12,139 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:13,139 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:14,142 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:15,108 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:15,143 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:16,145 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:17,144 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:17,782 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:49:18,102 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:49:18,103 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:49:18,149 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:19,144 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:20,145 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:20,176 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:21,149 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:22,149 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:22,372 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:49:22,373 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:49:22,373 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:49:22,740 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:49:22,740 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:49:22,989 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/liwo1zvs-events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:49:23,148 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:24,152 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:24,741 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:25,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:25,277 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:26,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:27,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:28,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:28,742 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:29,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:30,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:30,743 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:30,990 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:31,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:32,150 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:33,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:49:33,098 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:49:33,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:34,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:35,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:36,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:36,239 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:37,155 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:38,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:39,151 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:40,152 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:41,152 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:42,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:42,152 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:43,152 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:44,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:45,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:46,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:47,110 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:47,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:47,783 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:49:48,097 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:49:48,098 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:49:48,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:49,153 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:50,154 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:51,154 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:52,154 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:52,212 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:53,173 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:54,156 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:54,281 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:49:54,281 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 with policy now |
| 2025-01-01 11:49:54,289 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:49:54,290 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:49:54,290 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:49:54,291 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:49:54,578 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/qhh694eq-media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 |
| 2025-01-01 11:49:54,751 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:49:54,751 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:54,751 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 |
| 2025-01-01 11:49:54,752 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval |
| 2025-01-01 11:49:55,008 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/18v4tl5l-media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 |
| 2025-01-01 11:49:55,160 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:56,175 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:56,752 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:49:57,165 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:57,323 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:49:58,166 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:49:59,168 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:00,169 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:00,753 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:01,173 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:02,168 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:02,754 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:03,101 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:50:03,102 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:50:03,167 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:03,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:04,166 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:05,175 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:06,177 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:07,178 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:08,172 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:09,121 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:09,178 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:10,179 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:11,180 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:12,182 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:13,185 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:14,122 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:14,187 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:15,189 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:16,188 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:17,186 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:17,783 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:50:18,137 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:50:18,138 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:50:18,189 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:18,439 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:50:18,439 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:50:18,439 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:50:18,759 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:50:18,759 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:50:19,155 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:19,197 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:20,192 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:20,760 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:21,198 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:22,194 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:23,217 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:24,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:24,440 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:24,761 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:25,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:26,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:27,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:28,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:29,193 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:30,114 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:30,194 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:31,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:32,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:33,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:50:33,099 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:50:33,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:34,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:35,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:35,254 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:36,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:37,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:38,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:38,766 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:39,195 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:40,196 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:41,118 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:41,196 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:42,196 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:43,196 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:44,196 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:45,199 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:46,197 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:47,118 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:47,197 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:47,784 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:50:48,099 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:50:48,099 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:50:48,197 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:49,198 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:50,198 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:51,204 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:51,529 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:50:51,529 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 with policy now |
| 2025-01-01 11:50:51,535 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:50:51,536 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:50:51,536 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:50:51,537 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:50:51,770 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:50:51,770 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 |
| 2025-01-01 11:50:51,771 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval |
| 2025-01-01 11:50:51,790 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/0iu5p2o0-media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 |
| 2025-01-01 11:50:52,050 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/j32q2l82-media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 |
| 2025-01-01 11:50:52,202 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:52,556 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:52,771 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:53,206 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:53,771 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:54,214 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:55,206 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:56,213 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:56,772 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:50:57,203 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:57,605 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:50:58,225 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:50:59,216 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:00,215 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:00,773 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:01,204 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:02,209 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:03,116 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:51:03,116 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:51:03,208 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:51:03,209 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:04,211 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:05,222 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:06,222 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:07,222 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:08,214 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:08,499 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:51:08,499 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:08,499 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:51:08,499 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:08,776 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:51:08,776 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:51:08,776 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:08,913 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 11:51:08,913 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 11:51:09,061 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/pma5ojbp-events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:51:09,120 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:51:09,215 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:09,425 INFO wandb-upload_0:1797655 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1b9puqx6 |
| 2025-01-01 11:51:09,551 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 11:51:09,559 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table |
| 2025-01-01 11:51:09,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table/eval |
| 2025-01-01 11:51:09,952 INFO SenderThread:1797655 [sender.py:send_request_log_artifact():1525] logged artifact run-8511ppcj-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDcwMzQ5OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTI0NDQ0Mg==', 'latestArtifact': None}} |
| 2025-01-01 11:51:09,952 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:51:09,952 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-01 11:51:09,953 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:51:09,953 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,954 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,954 DEBUG SenderThread:1797655 [sender.py:send():379] send: telemetry |
| 2025-01-01 11:51:09,954 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,958 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,958 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,958 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,958 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,959 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,959 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,959 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,959 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,960 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,960 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,960 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,960 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,961 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,961 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,961 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,961 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,961 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,962 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,962 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,962 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:09,962 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:09,963 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 11:51:10,126 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/7hxr9auz-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:51:10,167 INFO wandb-upload_1:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/79iihm2m-media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json |
| 2025-01-01 11:51:10,215 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:10,332 INFO wandb-upload_0:1797655 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp7k2ccjgz |
| 2025-01-01 11:51:10,371 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/tzw4ycwg-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:51:10,713 INFO SenderThread:1797655 [sender.py:send_request_log_artifact():1525] logged artifact run-8511ppcj-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDcwMzUyMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTI0NDQ0Nw==', 'latestArtifact': None}} |
| 2025-01-01 11:51:10,714 DEBUG SenderThread:1797655 [sender.py:send():379] send: files |
| 2025-01-01 11:51:10,714 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json with policy now |
| 2025-01-01 11:51:10,714 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:51:10,714 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:10,715 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:10,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:51:10,777 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:10,984 INFO wandb-upload_1:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/cn7k02se-media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json |
| 2025-01-01 11:51:11,215 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:11,721 DEBUG SenderThread:1797655 [sender.py:send():379] send: exit |
| 2025-01-01 11:51:11,722 INFO SenderThread:1797655 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-01 11:51:11,722 INFO SenderThread:1797655 [sender.py:send_exit():588] handling runtime: 293 |
| 2025-01-01 11:51:11,722 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:11,722 INFO SenderThread:1797655 [sender.py:send_exit():594] send defer |
| 2025-01-01 11:51:11,722 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:11,722 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-01 11:51:11,722 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:11,722 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-01 11:51:11,723 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-01 11:51:11,723 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:11,723 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-01 11:51:11,723 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:11,723 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-01 11:51:11,723 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-01 11:51:11,723 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:11,723 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-01 11:51:11,723 INFO HandlerThread:1797655 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-01 11:51:11,723 DEBUG SystemMonitor:1797655 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-01 11:51:11,723 DEBUG SystemMonitor:1797655 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-01 11:51:11,724 INFO HandlerThread:1797655 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-01 11:51:11,724 INFO HandlerThread:1797655 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-01 11:51:11,777 INFO HandlerThread:1797655 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-01 11:51:11,777 INFO HandlerThread:1797655 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-01 11:51:11,777 INFO HandlerThread:1797655 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-01 11:51:11,778 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:11,778 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-01 11:51:11,778 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-01 11:51:11,778 DEBUG SenderThread:1797655 [sender.py:send():379] send: stats |
| 2025-01-01 11:51:11,778 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:11,778 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-01 11:51:11,778 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:11,778 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-01 11:51:11,778 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-01 11:51:11,779 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:11,779 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-01 11:51:11,779 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:51:12,780 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:18,504 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:51:18,504 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:18,505 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:51:18,505 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-01 11:51:18,505 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-01 11:51:18,505 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-01 11:51:18,506 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:51:18,506 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:18,506 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-01 11:51:18,506 DEBUG SenderThread:1797655 [sender.py:send():379] send: history |
| 2025-01-01 11:51:18,507 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:51:18,507 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:18,507 DEBUG SenderThread:1797655 [sender.py:send():379] send: summary |
| 2025-01-01 11:51:18,508 INFO SenderThread:1797655 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:51:18,508 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:18,508 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-01 11:51:18,508 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-01 11:51:18,508 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:18,508 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-01 11:51:18,509 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:18,509 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-01 11:51:18,512 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:51:18,635 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-01 11:51:18,636 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:18,636 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-01 11:51:18,636 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:18,636 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-01 11:51:18,723 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:51:18,782 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:51:18,782 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/config.yaml |
| 2025-01-01 11:51:21,623 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-01 11:51:21,623 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:51:21,624 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:21,624 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-01 11:51:21,624 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:21,624 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-01 11:51:21,624 INFO SenderThread:1797655 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-01 11:51:21,626 INFO SenderThread:1797655 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-01 11:51:21,628 INFO SenderThread:1797655 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-01 11:51:21,637 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-01 11:51:21,637 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:21,637 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-01 11:51:21,637 DEBUG SenderThread:1797655 [sender.py:send():379] send: artifact |
| 2025-01-01 11:51:21,724 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:51:21,783 INFO Thread-12 :1797655 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:21,816 INFO SenderThread:1797655 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'versionIndex': 404}}} |
| 2025-01-01 11:51:21,816 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:21,816 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-01 11:51:21,816 INFO SenderThread:1797655 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-01 11:51:22,784 INFO SenderThread:1797655 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files |
| 2025-01-01 11:51:22,784 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/config.yaml config.yaml |
| 2025-01-01 11:51:22,784 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json wandb-summary.json |
| 2025-01-01 11:51:22,784 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/requirements.txt requirements.txt |
| 2025-01-01 11:51:22,785 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-01 11:51:22,785 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log output.log |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/diff.patch diff.patch |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/0_eval_0_28199d60beae23009631.mp4 media/videos/eval/0_eval_0_28199d60beae23009631.mp4 |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 media/videos/eval/4_eval_4_77c14f84a9e39e1d85ee.mp4 |
| 2025-01-01 11:51:22,790 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 media/videos/eval/1_eval_1_c50442085fb847ef468d.mp4 |
| 2025-01-01 11:51:22,791 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 media/videos/eval/3_eval_3_c9555eba9b6b5e518c21.mp4 |
| 2025-01-01 11:51:22,791 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 media/videos/eval/2_eval_2_07e03518140cc7a8fdab.mp4 |
| 2025-01-01 11:51:22,793 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json media/table/eval/total_success_rate_ci_6_101bd877386d563fcdb4.table.json |
| 2025-01-01 11:51:22,794 INFO SenderThread:1797655 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:51:22,794 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-01 11:51:22,794 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:51:22,794 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:22,794 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-01 11:51:22,794 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:22,794 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-01 11:51:22,794 INFO SenderThread:1797655 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 11:51:22,968 INFO wandb-upload_0:1797655 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/wandb-summary.json |
| 2025-01-01 11:51:22,984 INFO wandb-upload_2:1797655 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/config.yaml |
| 2025-01-01 11:51:22,995 INFO wandb-upload_1:1797655 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/requirements.txt |
| 2025-01-01 11:51:22,997 INFO wandb-upload_4:1797655 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/files/output.log |
| 2025-01-01 11:51:23,093 INFO wandb-upload_3:1797655 [upload_job.py:push():130] Uploaded file /tmp/tmp42gfm2n1wandb/p1bqef6k-events.out.tfevents.1735749978.leonmkim-ROG-Strix-G15CS-G15CS.1797490.0 |
| 2025-01-01 11:51:23,294 INFO Thread-11 (_thread_body):1797655 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-01 11:51:23,294 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:23,294 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-01 11:51:23,294 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:23,294 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-01 11:51:23,294 INFO SenderThread:1797655 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-01 11:51:23,295 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-01 11:51:23,296 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:23,296 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-01 11:51:23,296 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:23,296 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-01 11:51:23,296 INFO SenderThread:1797655 [file_stream.py:finish():601] file stream finish called |
| 2025-01-01 11:51:23,533 INFO SenderThread:1797655 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-01 11:51:23,533 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-01 11:51:23,534 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:23,534 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-01 11:51:23,534 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:23,534 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-01 11:51:23,534 INFO SenderThread:1797655 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-01 11:51:23,534 DEBUG SenderThread:1797655 [sender.py:send():379] send: final |
| 2025-01-01 11:51:23,534 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:51:23,534 INFO HandlerThread:1797655 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-01 11:51:23,534 DEBUG SenderThread:1797655 [sender.py:send():379] send: footer |
| 2025-01-01 11:51:23,535 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:51:23,535 INFO SenderThread:1797655 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-01 11:51:23,535 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:51:23,535 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-01 11:51:23,535 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:51:23,536 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:51:23,536 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-01 11:51:23,536 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: server_info |
| 2025-01-01 11:51:23,536 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-01 11:51:23,538 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:51:23,586 DEBUG SenderThread:1797655 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:51:23,586 INFO MainThread:1797655 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-01 11:51:23,586 INFO MainThread:1797655 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-01 11:51:23,587 INFO MainThread:1797655 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-01 11:51:23,587 DEBUG HandlerThread:1797655 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-01 11:51:23,587 INFO HandlerThread:1797655 [handler.py:finish():882] shutting down handler |
| 2025-01-01 11:51:24,536 INFO WriterThread:1797655 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/8511ppcj/wandb/run-20250101_114617-8511ppcj/run-8511ppcj.wandb |
| 2025-01-01 11:51:24,586 INFO SenderThread:1797655 [sender.py:finish():1615] shutting down sender |
| 2025-01-01 11:51:24,586 INFO SenderThread:1797655 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 11:51:24,586 INFO SenderThread:1797655 [file_pusher.py:join():175] waiting for file pusher |
|
|
