| 2024-12-29 21:25:12,629 INFO StreamThr :1360479 [internal.py:wandb_internal():85] W&B internal server running at pid: 1360479, started at: 2024-12-29 21:25:12.629472 |
| 2024-12-29 21:25:12,631 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status |
| 2024-12-29 21:25:12,641 INFO WriterThread:1360479 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/run-oco0rjll.wandb |
| 2024-12-29 21:25:12,641 DEBUG SenderThread:1360479 [sender.py:send():379] send: header |
| 2024-12-29 21:25:12,651 DEBUG SenderThread:1360479 [sender.py:send():379] send: run |
| 2024-12-29 21:25:12,989 INFO SenderThread:1360479 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files |
| 2024-12-29 21:25:12,989 INFO SenderThread:1360479 [sender.py:_start_run_threads():1188] run started: oco0rjll with start time 1735525512.630035 |
| 2024-12-29 21:25:13,005 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-29 21:25:13,006 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: check_version |
| 2024-12-29 21:25:13,088 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-29 21:25:13,093 DEBUG HandlerThread:1360479 [system_info.py:__init__():26] System info init |
| 2024-12-29 21:25:13,093 DEBUG HandlerThread:1360479 [system_info.py:__init__():41] System info init done |
| 2024-12-29 21:25:13,093 INFO HandlerThread:1360479 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-29 21:25:13,094 INFO SystemMonitor:1360479 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-29 21:25:13,094 INFO HandlerThread:1360479 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-29 21:25:13,094 INFO SystemMonitor:1360479 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-29 21:25:13,095 INFO SystemMonitor:1360479 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-29 21:25:13,095 INFO SystemMonitor:1360479 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-29 21:25:13,096 INFO SystemMonitor:1360479 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-29 21:25:13,096 INFO SystemMonitor:1360479 [interfaces.py:start():188] Started network monitoring |
| 2024-12-29 21:25:13,109 DEBUG HandlerThread:1360479 [system_info.py:probe():152] Probing system |
| 2024-12-29 21:25:13,110 DEBUG HandlerThread:1360479 [system_info.py:_probe_git():137] Probing git |
| 2024-12-29 21:25:13,115 DEBUG HandlerThread:1360479 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-29 21:25:13,115 DEBUG HandlerThread:1360479 [system_info.py:probe():200] Probing system done |
| 2024-12-29 21:25:13,115 DEBUG HandlerThread:1360479 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-30T02:25:13.109189', 'startedAt': '2024-12-30T02:25:12.622611', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '23581857e509febdae7f0ac45cc9cf4f9081f85c'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3762.1391249999997, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4520.366, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4546.057, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 477.0821304321289}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-29 21:25:13,115 INFO HandlerThread:1360479 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-29 21:25:13,115 INFO HandlerThread:1360479 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-29 21:25:13,115 DEBUG HandlerThread:1360479 [system_info.py:_save_code():44] Saving code |
| 2024-12-29 21:25:13,116 DEBUG HandlerThread:1360479 [system_info.py:_save_code():67] Saving code done |
| 2024-12-29 21:25:13,116 DEBUG HandlerThread:1360479 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-29 21:25:13,179 DEBUG HandlerThread:1360479 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-29 21:25:13,179 INFO HandlerThread:1360479 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-29 21:25:13,182 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:25:13,182 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-29 21:25:13,183 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-29 21:25:13,183 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-29 21:25:13,418 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-29 21:25:13,418 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-29 21:25:13,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:25:13,419 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:25:13,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:13,422 INFO HandlerThread:1360479 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/tb" |
| save: true |
| } |
| control { |
| relay_id: "371900e530ae4f619f94a7297cc74e9c" |
| } |
| _info { |
| stream_id: "oco0rjll" |
| } |
|
|
| 2024-12-29 21:25:13,422 DEBUG HandlerThread:1360479 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-29 21:25:13,452 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/97vjqa7c-diff.patch |
| 2024-12-29 21:25:13,476 INFO wandb-upload_1:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/1k2sknsw-code/FISH/eval_robot.py |
| 2024-12-29 21:25:13,479 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/qurbt2cg-wandb-metadata.json |
| 2024-12-29 21:25:13,521 DEBUG SenderThread:1360479 [sender.py:send():379] send: telemetry |
| 2024-12-29 21:25:13,521 DEBUG SenderThread:1360479 [sender.py:send():379] send: tbrecord |
| 2024-12-29 21:25:13,522 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:25:13,522 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 with policy live |
| 2024-12-29 21:25:13,710 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/f1ksd97h-events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:25:13,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/requirements.txt |
| 2024-12-29 21:25:13,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/code/FISH/eval_robot.py |
| 2024-12-29 21:25:13,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:13,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/diff.patch |
| 2024-12-29 21:25:13,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-metadata.json |
| 2024-12-29 21:25:13,992 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:25:13,992 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/code |
| 2024-12-29 21:25:13,992 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/code/FISH |
| 2024-12-29 21:25:14,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:15,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:15,990 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:16,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:17,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:17,663 DEBUG SenderThread:1360479 [sender.py:send():379] send: config |
| 2024-12-29 21:25:17,663 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:18,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:19,421 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:19,991 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:20,421 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:21,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:22,433 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:23,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:23,421 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:23,992 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:24,421 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:25,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:26,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:27,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:27,993 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:28,418 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:25:28,419 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:25:28,461 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:28,526 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:29,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:30,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:31,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:32,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:33,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:34,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:34,423 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:35,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:36,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:37,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:37,996 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:38,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:39,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:39,997 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:40,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:40,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:41,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:41,997 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:42,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:43,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:25:43,419 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:25:43,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:43,998 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:44,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:45,424 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:45,603 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:45,998 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:45,999 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/config.yaml |
| 2024-12-29 21:25:46,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:47,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:47,999 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:48,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:49,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:49,999 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:50,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:50,804 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:51,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:52,000 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:52,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:53,425 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:54,001 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:54,426 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:55,426 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:55,997 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:25:56,001 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:56,426 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:57,426 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:58,002 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:25:58,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:25:58,419 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:25:58,426 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:25:59,427 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:00,002 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:00,427 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:01,003 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:01,428 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:02,004 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:02,428 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:03,428 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:04,004 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:04,428 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:05,428 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:06,005 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:06,194 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:06,429 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:07,429 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:08,005 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:08,429 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:08,601 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-29 21:26:08,602 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:26:08,602 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 with policy now |
| 2024-12-29 21:26:08,602 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:26:08,602 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:26:08,603 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:26:08,857 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/r82fuotw-media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 |
| 2024-12-29 21:26:09,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:26:09,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 |
| 2024-12-29 21:26:09,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos |
| 2024-12-29 21:26:09,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval |
| 2024-12-29 21:26:09,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media |
| 2024-12-29 21:26:09,266 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/4nfn6llk-media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 |
| 2024-12-29 21:26:09,429 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:10,006 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:10,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:11,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:11,603 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:12,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:13,097 DEBUG SystemMonitor:1360479 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-29 21:26:13,097 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:26:13,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:26:13,421 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:26:13,461 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:14,007 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:14,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:15,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:16,008 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:16,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:16,934 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:17,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:18,009 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:26:18,259 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/znjalm43-events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:26:18,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:19,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:20,010 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:20,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:21,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:21,965 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:22,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:23,431 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:24,011 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:24,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:25,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:26,012 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:26,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:27,006 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:27,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:28,012 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:28,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:26:28,419 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:26:28,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:29,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:30,013 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:30,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:31,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:32,014 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:32,202 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:32,432 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:33,433 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:34,015 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:34,433 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:35,433 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:36,015 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:36,434 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:37,204 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:37,434 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:38,016 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:38,434 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:39,434 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:40,017 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:40,434 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:41,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:42,018 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:42,396 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:42,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:43,098 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:26:43,419 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:26:43,420 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:26:43,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:44,018 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:44,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:45,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:46,019 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:46,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:47,397 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:47,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:48,019 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:48,435 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:49,437 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:49,687 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:26:49,687 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 with policy now |
| 2024-12-29 21:26:49,688 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-29 21:26:49,688 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:26:49,689 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:26:49,689 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:26:49,933 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/epacwhln-media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 |
| 2024-12-29 21:26:50,020 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:26:50,020 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:50,021 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 |
| 2024-12-29 21:26:50,021 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval |
| 2024-12-29 21:26:50,292 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/uuduyhwz-media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 |
| 2024-12-29 21:26:50,438 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:51,451 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:52,021 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:52,447 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:52,709 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:53,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:54,446 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:55,445 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:56,022 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:56,440 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:57,440 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:58,023 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:26:58,133 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:26:58,437 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:26:58,438 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:26:58,440 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:26:58,807 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:26:58,807 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:26:58,812 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:26:59,023 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:26:59,023 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:26:59,449 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:00,023 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:00,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:01,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:02,024 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:02,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:03,225 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:03,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:04,025 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:04,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:05,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:06,026 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:06,441 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:07,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:08,027 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:08,238 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:08,481 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:09,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:10,027 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:10,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:11,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:12,028 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:12,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:13,098 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:27:13,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:27:13,420 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:27:13,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:13,532 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:14,029 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:14,442 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:15,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:16,030 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:16,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:17,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:18,030 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:18,485 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:18,642 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:19,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:20,031 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:20,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:21,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:22,032 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:22,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:23,481 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:23,837 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:24,032 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:24,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:25,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:26,033 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:26,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:27,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:28,034 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:28,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:27:28,421 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:27:28,444 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:29,035 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:29,456 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:30,035 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:30,357 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:27:30,358 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 with policy now |
| 2024-12-29 21:27:30,363 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-29 21:27:30,364 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:27:30,364 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:27:30,365 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:27:30,456 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:30,610 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/of9cesez-media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 |
| 2024-12-29 21:27:31,035 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:27:31,035 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 |
| 2024-12-29 21:27:31,035 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval |
| 2024-12-29 21:27:31,310 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/p7ulzqh4-media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 |
| 2024-12-29 21:27:31,457 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:32,035 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:32,452 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:33,452 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:34,073 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:34,454 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:35,455 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:36,037 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:36,466 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:37,457 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:38,037 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:38,453 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:39,445 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:39,453 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:40,456 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:41,457 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:42,465 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:43,099 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:27:43,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:27:43,421 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:27:43,466 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:44,468 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:44,552 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:45,040 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:27:45,286 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/w7589ncm-events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:27:45,454 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:45,649 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:27:45,649 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:27:45,649 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:27:46,040 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:27:46,040 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:46,455 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:47,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:48,041 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:48,467 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:49,458 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:49,834 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:50,041 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:50,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:51,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:52,042 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:52,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:53,497 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:54,043 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:54,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:54,870 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:27:55,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:56,043 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:56,459 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:57,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:58,044 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:27:58,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:27:58,420 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:27:58,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:59,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:27:59,941 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:00,045 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:00,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:01,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:02,046 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:02,460 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:03,501 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:04,046 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:04,462 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:05,062 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:05,462 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:06,047 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:06,462 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:07,462 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:08,048 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:08,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:09,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:10,049 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:10,256 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:10,462 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:11,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:12,049 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:12,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:13,099 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:28:13,420 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:28:13,420 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:28:13,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:14,050 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:14,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:15,263 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:15,463 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:16,051 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:16,472 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:17,233 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:28:17,234 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 with policy now |
| 2024-12-29 21:28:17,238 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-29 21:28:17,239 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:28:17,239 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:28:17,239 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:28:17,466 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:17,502 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/09zme4ma-media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 |
| 2024-12-29 21:28:18,051 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:28:18,052 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:18,052 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 |
| 2024-12-29 21:28:18,052 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval |
| 2024-12-29 21:28:18,332 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/8rm19avh-media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 |
| 2024-12-29 21:28:18,475 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:19,477 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:20,457 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:20,481 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:21,477 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:22,053 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:22,489 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:23,483 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:24,483 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:25,481 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:25,785 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:26,054 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:26,485 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:27,483 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:28,430 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:28:28,430 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:28:28,489 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:29,482 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:30,481 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:31,483 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:31,580 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:32,490 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:33,487 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:33,701 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:28:33,701 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:28:33,701 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:28:34,057 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:28:34,057 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:34,057 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:28:34,492 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:35,493 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:36,057 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:36,502 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:37,009 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:37,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:38,058 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:38,541 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:39,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:40,059 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:40,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:41,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:42,060 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:42,119 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:42,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:43,100 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:28:43,427 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:28:43,427 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:28:43,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:44,060 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:44,507 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:45,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:46,061 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:46,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:47,125 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:47,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:48,062 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:48,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:49,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:50,063 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:50,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:51,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:52,064 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:52,130 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:52,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:53,508 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:54,064 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:54,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:55,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:56,065 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:56,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:57,322 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:28:57,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:58,066 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:28:58,427 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:28:58,428 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:28:58,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:28:59,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:00,067 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:00,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:01,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:02,067 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:02,509 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:02,524 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:29:03,510 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:04,068 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:04,510 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:05,517 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:05,589 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-29 21:29:05,589 DEBUG SenderThread:1360479 [sender.py:send():379] send: files |
| 2024-12-29 21:29:05,589 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 with policy now |
| 2024-12-29 21:29:05,590 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:29:05,590 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:29:05,590 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:29:05,834 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/6o3pw2m5-media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 |
| 2024-12-29 21:29:06,069 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:29:06,069 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:06,069 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 |
| 2024-12-29 21:29:06,069 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval |
| 2024-12-29 21:29:06,324 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/njaqymsx-media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 |
| 2024-12-29 21:29:06,517 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:07,528 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:07,611 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:29:08,069 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:08,520 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:09,520 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:10,070 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:10,538 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:11,525 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:12,530 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:13,100 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:29:13,101 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:29:13,443 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-29 21:29:13,444 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-29 21:29:13,549 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:14,072 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:14,539 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:14,706 DEBUG SenderThread:1360479 [sender.py:send():379] send: exit |
| 2024-12-29 21:29:14,706 INFO SenderThread:1360479 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-29 21:29:14,706 INFO SenderThread:1360479 [sender.py:send_exit():588] handling runtime: 241 |
| 2024-12-29 21:29:14,721 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:29:14,721 INFO SenderThread:1360479 [sender.py:send_exit():594] send defer |
| 2024-12-29 21:29:14,721 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:14,721 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-29 21:29:14,722 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:14,722 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-29 21:29:14,722 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-29 21:29:14,722 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:14,722 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-29 21:29:14,722 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:14,722 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-29 21:29:14,722 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-29 21:29:14,722 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:14,722 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-29 21:29:14,722 INFO HandlerThread:1360479 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-29 21:29:14,722 DEBUG SystemMonitor:1360479 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-29 21:29:14,722 INFO HandlerThread:1360479 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-29 21:29:14,722 DEBUG SystemMonitor:1360479 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-29 21:29:14,723 INFO HandlerThread:1360479 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-29 21:29:14,773 INFO HandlerThread:1360479 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-29 21:29:14,774 INFO HandlerThread:1360479 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-29 21:29:14,774 INFO HandlerThread:1360479 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-29 21:29:14,774 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:14,774 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-29 21:29:14,774 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-29 21:29:14,775 DEBUG SenderThread:1360479 [sender.py:send():379] send: stats |
| 2024-12-29 21:29:14,775 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:14,775 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:29:14,775 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-29 21:29:14,775 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:29:14,796 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:29:14,797 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:14,797 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-29 21:29:14,797 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-29 21:29:14,797 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:14,797 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-29 21:29:15,072 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:29:15,072 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:29:15,347 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /tmp/tmpkjk2br4wwandb/mr7i954z-events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:29:16,072 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:21,759 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:21,759 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:29:21,759 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:21,759 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-29 21:29:21,759 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-29 21:29:21,759 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-29 21:29:21,759 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:21,760 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:21,760 DEBUG SenderThread:1360479 [sender.py:send():379] send: history |
| 2024-12-29 21:29:21,760 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-29 21:29:21,760 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-29 21:29:21,760 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:29:21,760 DEBUG SenderThread:1360479 [sender.py:send():379] send: summary |
| 2024-12-29 21:29:21,761 INFO SenderThread:1360479 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-29 21:29:21,761 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:21,761 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-29 21:29:21,761 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-29 21:29:21,761 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:21,761 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-29 21:29:21,761 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:21,761 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-29 21:29:21,761 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-29 21:29:21,761 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-29 21:29:21,761 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:21,762 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-29 21:29:21,762 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:21,762 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-29 21:29:22,074 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:29:22,707 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:23,721 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-29 21:29:23,721 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:23,721 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:23,721 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-29 21:29:23,722 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:23,722 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-29 21:29:23,722 INFO SenderThread:1360479 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-29 21:29:23,724 INFO SenderThread:1360479 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-29 21:29:23,738 INFO SenderThread:1360479 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-29 21:29:23,752 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-29 21:29:23,753 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:23,753 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-29 21:29:23,753 DEBUG SenderThread:1360479 [sender.py:send():379] send: artifact |
| 2024-12-29 21:29:24,075 INFO Thread-12 :1360479 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:24,192 INFO wandb-upload_1:1360479 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpu0f2y331 |
| 2024-12-29 21:29:24,192 INFO wandb-upload_0:1360479 [upload_job.py:push():85] Skipped uploading /tmp/tmp6dzmspxx/wandb-job.json |
| 2024-12-29 21:29:24,336 INFO wandb-upload_2:1360479 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpf1m3m3qp |
| 2024-12-29 21:29:24,708 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:24,896 INFO SenderThread:1360479 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxMjA3MTYzNw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxMjA2MzIwOA==', 'versionIndex': 387}}} |
| 2024-12-29 21:29:24,896 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:24,896 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-29 21:29:24,896 INFO SenderThread:1360479 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-29 21:29:25,076 INFO SenderThread:1360479 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files |
| 2024-12-29 21:29:25,076 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/config.yaml config.yaml |
| 2024-12-29 21:29:25,076 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json wandb-summary.json |
| 2024-12-29 21:29:25,077 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/requirements.txt requirements.txt |
| 2024-12-29 21:29:25,078 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-29 21:29:25,078 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 events.out.tfevents.1735525513.leonmkim-ROG-Strix-G15CS-G15CS.1360290.0 |
| 2024-12-29 21:29:25,079 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log output.log |
| 2024-12-29 21:29:25,079 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/diff.patch diff.patch |
| 2024-12-29 21:29:25,083 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 media/videos/eval/1_eval_1_df2b523b5c69376abbb6.mp4 |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 media/videos/eval/2_eval_2_56287c330ba45a21ceae.mp4 |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 media/videos/eval/4_eval_4_277cb7c2f38e9f60cb6a.mp4 |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 media/videos/eval/3_eval_3_d9096d3d3b1128c5f3af.mp4 |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 media/videos/eval/0_eval_0_7f4ade8f7a09940de9a3.mp4 |
| 2024-12-29 21:29:25,084 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-29 21:29:25,084 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:25,084 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:25,084 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-29 21:29:25,085 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:25,085 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-29 21:29:25,085 INFO SenderThread:1360479 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-29 21:29:25,270 INFO wandb-upload_2:1360479 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/wandb-summary.json |
| 2024-12-29 21:29:25,292 INFO wandb-upload_3:1360479 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/output.log |
| 2024-12-29 21:29:25,297 INFO wandb-upload_1:1360479 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/config.yaml |
| 2024-12-29 21:29:25,317 INFO wandb-upload_0:1360479 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/files/requirements.txt |
| 2024-12-29 21:29:25,517 INFO Thread-11 (_thread_body):1360479 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-29 21:29:25,517 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:25,517 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-29 21:29:25,517 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:25,517 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-29 21:29:25,517 INFO SenderThread:1360479 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-29 21:29:25,518 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-29 21:29:25,518 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:25,518 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-29 21:29:25,518 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:25,519 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-29 21:29:25,519 INFO SenderThread:1360479 [file_stream.py:finish():601] file stream finish called |
| 2024-12-29 21:29:25,708 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:25,725 INFO SenderThread:1360479 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-29 21:29:25,725 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-29 21:29:25,725 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:25,725 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:25,725 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-29 21:29:25,725 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:25,725 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-29 21:29:25,725 INFO SenderThread:1360479 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-29 21:29:25,726 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-29 21:29:25,726 DEBUG SenderThread:1360479 [sender.py:send():379] send: final |
| 2024-12-29 21:29:25,726 INFO HandlerThread:1360479 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-29 21:29:25,726 DEBUG SenderThread:1360479 [sender.py:send():379] send: footer |
| 2024-12-29 21:29:25,726 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: defer |
| 2024-12-29 21:29:25,726 INFO SenderThread:1360479 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-29 21:29:25,726 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:25,726 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:25,727 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-29 21:29:25,727 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: server_info |
| 2024-12-29 21:29:25,728 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-29 21:29:25,728 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-29 21:29:25,729 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-29 21:29:25,729 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-29 21:29:25,782 DEBUG SenderThread:1360479 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-29 21:29:25,783 INFO MainThread:1360479 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-29 21:29:25,783 INFO MainThread:1360479 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-29 21:29:25,783 INFO MainThread:1360479 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-29 21:29:25,783 DEBUG HandlerThread:1360479 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-29 21:29:25,783 INFO HandlerThread:1360479 [handler.py:finish():882] shutting down handler |
| 2024-12-29 21:29:26,728 INFO WriterThread:1360479 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1000_0/212506/wandb/run-20241229_212512-oco0rjll/run-oco0rjll.wandb |
| 2024-12-29 21:29:26,783 INFO SenderThread:1360479 [sender.py:finish():1615] shutting down sender |
| 2024-12-29 21:29:26,783 INFO SenderThread:1360479 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-29 21:29:26,783 INFO SenderThread:1360479 [file_pusher.py:join():175] waiting for file pusher |
|
|
