| 2025-01-06 22:18:10,159 INFO StreamThr :2123564 [internal.py:wandb_internal():85] W&B internal server running at pid: 2123564, started at: 2025-01-06 22:18:10.159088 |
| 2025-01-06 22:18:10,160 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 22:18:10,172 INFO WriterThread:2123564 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/run-917c47lf.wandb |
| 2025-01-06 22:18:10,172 DEBUG SenderThread:2123564 [sender.py:send():379] send: header |
| 2025-01-06 22:18:10,181 DEBUG SenderThread:2123564 [sender.py:send():379] send: run |
| 2025-01-06 22:18:10,412 INFO SenderThread:2123564 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files |
| 2025-01-06 22:18:10,413 INFO SenderThread:2123564 [sender.py:_start_run_threads():1188] run started: 917c47lf with start time 1736219890.1595 |
| 2025-01-06 22:18:10,429 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 22:18:10,429 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 22:18:10,499 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 22:18:10,504 DEBUG HandlerThread:2123564 [system_info.py:__init__():26] System info init |
| 2025-01-06 22:18:10,504 DEBUG HandlerThread:2123564 [system_info.py:__init__():41] System info init done |
| 2025-01-06 22:18:10,504 INFO HandlerThread:2123564 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 22:18:10,504 INFO SystemMonitor:2123564 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 22:18:10,504 INFO SystemMonitor:2123564 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 22:18:10,504 INFO HandlerThread:2123564 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 22:18:10,505 INFO SystemMonitor:2123564 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 22:18:10,506 INFO SystemMonitor:2123564 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 22:18:10,507 INFO SystemMonitor:2123564 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 22:18:10,507 INFO SystemMonitor:2123564 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 22:18:10,518 DEBUG HandlerThread:2123564 [system_info.py:probe():152] Probing system |
| 2025-01-06 22:18:10,520 DEBUG HandlerThread:2123564 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 22:18:10,524 DEBUG HandlerThread:2123564 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 22:18:10,524 DEBUG HandlerThread:2123564 [system_info.py:probe():200] Probing system done |
| 2025-01-06 22:18:10,524 DEBUG HandlerThread:2123564 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T03:18:10.518603', 'startedAt': '2025-01-07T03:18:10.150271', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.4985, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 4500.056, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 564.8245735168457}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 22:18:10,524 INFO HandlerThread:2123564 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 22:18:10,524 INFO HandlerThread:2123564 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 22:18:10,524 DEBUG HandlerThread:2123564 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 22:18:10,526 DEBUG HandlerThread:2123564 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 22:18:10,526 DEBUG HandlerThread:2123564 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 22:18:10,588 DEBUG HandlerThread:2123564 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 22:18:10,589 INFO HandlerThread:2123564 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 22:18:10,592 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:18:10,592 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 22:18:10,592 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 22:18:10,592 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 22:18:10,822 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 22:18:10,823 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 22:18:10,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:18:10,824 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/6tpjjiq2-wandb-metadata.json |
| 2025-01-06 22:18:10,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:10,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:18:10,826 INFO HandlerThread:2123564 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/tb" |
| save: true |
| } |
| control { |
| relay_id: "cef7d876c4ad4fd199c5eb1a9bd32925" |
| } |
| _info { |
| stream_id: "917c47lf" |
| } |
|
|
| 2025-01-06 22:18:10,826 DEBUG HandlerThread:2123564 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 22:18:10,851 INFO wandb-upload_2:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/kdr9sx7r-diff.patch |
| 2025-01-06 22:18:10,855 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/5sqz2j7o-code/FISH/eval_robot.py |
| 2025-01-06 22:18:10,976 DEBUG SenderThread:2123564 [sender.py:send():379] send: telemetry |
| 2025-01-06 22:18:10,976 DEBUG SenderThread:2123564 [sender.py:send():379] send: tbrecord |
| 2025-01-06 22:18:10,977 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:18:10,977 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 with policy live |
| 2025-01-06 22:18:11,156 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/v1x7seua-events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:18:11,414 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/requirements.txt |
| 2025-01-06 22:18:11,414 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:18:11,414 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/diff.patch |
| 2025-01-06 22:18:11,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:11,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-metadata.json |
| 2025-01-06 22:18:11,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/code/FISH/eval_robot.py |
| 2025-01-06 22:18:11,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/code |
| 2025-01-06 22:18:11,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/code/FISH |
| 2025-01-06 22:18:11,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:12,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:13,413 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:13,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:14,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:15,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:15,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:16,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:17,522 DEBUG SenderThread:2123564 [sender.py:send():379] send: config |
| 2025-01-06 22:18:17,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:18,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:19,415 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:19,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:20,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:20,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:21,416 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:21,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:22,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:23,416 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:23,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:24,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:25,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:18:25,823 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:18:25,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:25,972 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:26,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:27,417 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:27,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:28,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:29,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:30,825 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:30,979 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:31,418 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:31,826 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:32,826 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:33,419 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:33,826 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:34,826 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:35,420 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:18:35,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:36,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:36,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:37,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:38,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:39,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:40,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:18:40,823 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:18:40,869 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:41,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:41,930 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:42,422 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/config.yaml |
| 2025-01-06 22:18:42,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:43,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:44,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:45,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:46,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:47,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:47,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:48,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:49,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:50,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:51,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:52,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:52,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:53,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:54,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:55,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:18:55,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:18:55,869 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:56,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:57,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:57,962 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:18:58,829 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:18:59,829 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:00,836 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:01,662 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:19:01,662 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 with policy now |
| 2025-01-06 22:19:01,662 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:19:01,663 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:19:01,663 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:19:01,664 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:19:01,836 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:01,901 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/37axuafr-media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 |
| 2025-01-06 22:19:02,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 |
| 2025-01-06 22:19:02,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:19:02,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos |
| 2025-01-06 22:19:02,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval |
| 2025-01-06 22:19:02,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media |
| 2025-01-06 22:19:02,683 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/uc6srzzt-media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 |
| 2025-01-06 22:19:02,837 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:03,427 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:03,674 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:03,838 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:04,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:05,838 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:06,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:07,429 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:07,838 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:08,833 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:08,838 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:09,429 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:09,838 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:10,507 DEBUG SystemMonitor:2123564 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 22:19:10,508 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:19:10,823 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:19:10,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:19:10,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:11,430 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:19:11,666 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/mzeyzwh7-events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:19:11,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:12,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:13,431 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:13,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:14,346 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:14,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:15,432 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:15,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:16,839 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:17,432 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:17,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:18,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:19,829 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:19,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:20,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:21,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:22,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:23,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:24,840 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:25,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:19:25,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:19:25,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:25,905 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:26,841 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:27,841 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:28,841 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:29,841 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:30,842 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:31,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:31,842 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:32,842 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:33,842 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:34,842 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:35,438 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:35,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:36,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:36,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:37,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:38,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:39,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:40,508 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:19:40,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:19:40,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:19:40,869 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:41,844 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:42,088 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:42,576 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:19:42,577 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:19:42,577 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 with policy now |
| 2025-01-06 22:19:42,577 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:19:42,577 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:19:42,578 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:19:42,791 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/qpwc9feh-media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 |
| 2025-01-06 22:19:42,845 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:43,441 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:43,441 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:19:43,441 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 |
| 2025-01-06 22:19:43,441 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval |
| 2025-01-06 22:19:43,713 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/tnputsod-media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 |
| 2025-01-06 22:19:43,847 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:44,844 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:45,441 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:45,845 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:46,845 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:47,442 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:47,830 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:47,845 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:48,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:49,443 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:49,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:50,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:51,225 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:19:51,226 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:19:51,226 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:19:51,443 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:51,444 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:19:51,444 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:19:51,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:52,851 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:53,444 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:53,637 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:53,852 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:54,852 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:55,445 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:55,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:19:55,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:19:55,847 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:56,848 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:57,446 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:57,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:58,789 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:19:58,861 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:19:59,446 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:19:59,863 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:00,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:01,863 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:02,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:03,830 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:03,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:04,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:05,862 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:06,869 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:07,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:08,837 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:08,864 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:09,864 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:10,509 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:20:10,824 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:20:10,824 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:20:10,869 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:11,864 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:12,867 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:13,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:13,974 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:14,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:15,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:16,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:17,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:18,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:19,834 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:19,866 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:20,868 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:21,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:22,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:23,865 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:24,868 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:24,882 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:25,180 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:20:25,180 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 with policy now |
| 2025-01-06 22:20:25,182 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:20:25,183 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:20:25,183 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:20:25,183 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:20:25,397 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/xkkmrbaq-media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 |
| 2025-01-06 22:20:25,455 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:25,455 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:20:25,455 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 |
| 2025-01-06 22:20:25,455 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval |
| 2025-01-06 22:20:25,728 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/u12h7cjr-media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 |
| 2025-01-06 22:20:25,829 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:20:25,829 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:20:25,873 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:26,876 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:27,455 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:27,874 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:28,879 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:29,456 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:29,877 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:30,365 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:30,877 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:31,882 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:32,885 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:33,457 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:33,887 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:34,890 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:35,836 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:35,888 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:36,889 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:37,895 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:38,898 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:39,904 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:40,509 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:20:40,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:20:40,829 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:20:40,899 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:40,969 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:41,897 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:42,903 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:43,910 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:44,905 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:45,905 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:46,843 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:46,910 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:47,911 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:48,909 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:49,905 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:50,914 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:51,852 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:51,912 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:52,909 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:53,464 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:20:53,706 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/b8z1ithe-events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:20:53,909 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:54,075 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:20:54,075 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:20:54,076 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:20:54,464 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:20:54,914 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:55,464 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:55,835 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:20:55,835 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:20:55,912 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:56,913 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:56,926 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:20:57,915 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:58,913 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:20:59,466 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:20:59,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:00,917 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:01,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:02,835 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:02,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:03,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:04,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:05,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:06,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:07,835 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:07,918 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:08,919 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:09,919 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:10,510 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:21:10,827 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:21:10,827 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:21:10,919 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:11,919 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:12,886 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:12,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:13,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:14,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:15,471 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:15,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:16,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:17,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:18,836 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:18,921 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:19,922 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:20,922 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:21,922 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:22,922 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:23,836 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:23,922 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:24,924 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:25,474 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:25,608 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:21:25,608 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 with policy now |
| 2025-01-06 22:21:25,613 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:21:25,614 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:21:25,614 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:21:25,617 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:21:25,816 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/5k2hh8oi-media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 |
| 2025-01-06 22:21:25,835 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:21:25,835 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:21:25,929 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:26,475 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:21:26,475 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 |
| 2025-01-06 22:21:26,475 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval |
| 2025-01-06 22:21:26,728 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/jpkayfdj-media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 |
| 2025-01-06 22:21:26,935 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:27,475 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:27,930 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:28,928 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:28,933 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:29,932 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:30,936 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:31,477 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:31,936 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:32,935 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:33,477 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:33,938 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:34,121 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:34,937 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:35,941 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:36,934 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:37,943 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:38,946 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:39,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:39,949 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:40,511 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:21:40,844 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:21:40,844 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:21:40,946 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:41,948 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:42,135 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:21:42,135 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:21:42,135 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:21:42,481 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:21:42,481 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:21:42,948 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:43,481 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:43,971 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:44,966 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:45,384 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:45,952 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:46,952 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:47,482 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:47,952 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:48,956 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:49,955 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:50,844 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:50,953 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:51,954 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:52,953 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:53,953 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:54,961 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:55,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:21:55,828 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:21:55,962 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:55,979 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:21:56,962 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:57,486 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:21:57,965 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:58,962 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:21:59,963 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:00,963 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:01,844 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:01,963 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:02,964 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:03,964 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:04,964 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:05,968 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:06,846 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:06,966 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:07,966 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:08,965 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:09,965 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:10,511 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:22:10,828 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:22:10,828 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:22:10,967 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:11,490 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:11,880 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:11,967 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:12,968 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:13,976 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:14,598 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:22:14,598 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 with policy now |
| 2025-01-06 22:22:14,607 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:22:14,608 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:22:14,608 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:14,609 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:14,811 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/ky50jorc-media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 |
| 2025-01-06 22:22:14,982 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:15,492 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:15,492 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:15,492 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 |
| 2025-01-06 22:22:15,492 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval |
| 2025-01-06 22:22:15,758 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/ngi6k4n8-media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 |
| 2025-01-06 22:22:15,986 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:16,986 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:17,492 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:17,850 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:17,982 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:19,006 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:19,493 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:19,987 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:20,994 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:21,494 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:21,999 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:22,851 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:22,997 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:23,495 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:24,006 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:25,009 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:25,841 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 22:22:25,841 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 22:22:25,994 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:27,006 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:27,975 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:27,999 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:29,000 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:30,016 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:31,005 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:32,012 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:33,004 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:33,058 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 22:22:33,058 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 22:22:33,265 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:22:33,498 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:33,498 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:22:33,499 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-06 22:22:33,499 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media |
| 2025-01-06 22:22:33,499 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval |
| 2025-01-06 22:22:33,499 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table |
| 2025-01-06 22:22:33,510 INFO wandb-upload_1:2123564 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphynk_6q6 |
| 2025-01-06 22:22:33,670 INFO wandb-upload_2:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/85pm559e-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-06 22:22:33,708 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 22:22:33,719 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 22:22:33,747 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/nelgnvoq-events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:22:33,940 INFO SenderThread:2123564 [sender.py:send_request_log_artifact():1525] logged artifact run-917c47lf-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjU3Mjg1MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc0MDMwNA==', 'latestArtifact': None}} |
| 2025-01-06 22:22:33,940 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:22:33,940 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json with policy now |
| 2025-01-06 22:22:33,941 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:33,941 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:22:33,941 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,941 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,942 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:22:33,942 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,946 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,946 DEBUG SenderThread:2123564 [sender.py:send():379] send: telemetry |
| 2025-01-06 22:22:33,946 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,946 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,946 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,946 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,947 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,947 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,947 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,947 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,948 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,948 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,948 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,948 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,948 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,949 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,949 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,949 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,949 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,950 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,950 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,950 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,950 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:33,951 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:33,951 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 22:22:34,004 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:34,189 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/102le9fn-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-06 22:22:34,288 INFO wandb-upload_0:2123564 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpmh_9g91f |
| 2025-01-06 22:22:34,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:22:34,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:34,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json |
| 2025-01-06 22:22:34,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval |
| 2025-01-06 22:22:34,680 INFO SenderThread:2123564 [sender.py:send_request_log_artifact():1525] logged artifact run-917c47lf-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjU3Mjg4Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDc0MDMwOQ==', 'latestArtifact': None}} |
| 2025-01-06 22:22:34,680 DEBUG SenderThread:2123564 [sender.py:send():379] send: files |
| 2025-01-06 22:22:34,680 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json with policy now |
| 2025-01-06 22:22:34,680 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:22:34,680 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:34,681 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:34,852 INFO wandb-upload_2:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/k8zaf9nu-media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json |
| 2025-01-06 22:22:35,004 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:35,094 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/t645rmde-media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json |
| 2025-01-06 22:22:35,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:35,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:36,005 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:36,361 DEBUG SenderThread:2123564 [sender.py:send():379] send: exit |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:send_exit():588] handling runtime: 265 |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:send_exit():594] send defer |
| 2025-01-06 22:22:36,362 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:36,362 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 22:22:36,362 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 22:22:36,362 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 22:22:36,363 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:36,363 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 22:22:36,363 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:36,363 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 22:22:36,363 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 22:22:36,363 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:36,363 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 22:22:36,363 INFO HandlerThread:2123564 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 22:22:36,363 DEBUG SystemMonitor:2123564 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 22:22:36,363 DEBUG SystemMonitor:2123564 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 22:22:36,364 INFO HandlerThread:2123564 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 22:22:36,364 INFO HandlerThread:2123564 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 22:22:36,414 INFO HandlerThread:2123564 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 22:22:36,415 INFO HandlerThread:2123564 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 22:22:36,415 INFO HandlerThread:2123564 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 22:22:36,416 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:36,416 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 22:22:36,416 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 22:22:36,416 DEBUG SenderThread:2123564 [sender.py:send():379] send: stats |
| 2025-01-06 22:22:36,416 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:36,416 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 22:22:36,417 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:36,417 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 22:22:36,417 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 22:22:36,417 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:36,417 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 22:22:36,500 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:37,501 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:43,200 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:22:43,200 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:43,200 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 22:22:43,202 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:43,202 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 22:22:43,202 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 22:22:43,202 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:22:43,202 DEBUG SenderThread:2123564 [sender.py:send():379] send: history |
| 2025-01-06 22:22:43,202 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:43,202 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 22:22:43,202 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 22:22:43,203 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:43,203 DEBUG SenderThread:2123564 [sender.py:send():379] send: summary |
| 2025-01-06 22:22:43,207 INFO SenderThread:2123564 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 22:22:43,207 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:43,207 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 22:22:43,207 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 22:22:43,207 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:43,207 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 22:22:43,208 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:43,208 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 22:22:43,211 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 22:22:43,328 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 22:22:43,328 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:43,328 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 22:22:43,329 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:43,329 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 22:22:43,363 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:22:43,503 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/config.yaml |
| 2025-01-06 22:22:43,503 INFO Thread-12 :2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:46,639 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 22:22:46,639 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:22:46,640 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:46,640 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 22:22:46,640 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:46,640 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 22:22:46,640 INFO SenderThread:2123564 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 22:22:46,642 INFO SenderThread:2123564 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 22:22:46,644 INFO SenderThread:2123564 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 22:22:46,657 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 22:22:46,658 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:46,658 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 22:22:46,658 DEBUG SenderThread:2123564 [sender.py:send():379] send: artifact |
| 2025-01-06 22:22:46,874 INFO SenderThread:2123564 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkzNjUwMA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} |
| 2025-01-06 22:22:46,874 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:46,874 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 22:22:46,874 INFO SenderThread:2123564 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 22:22:47,364 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:22:47,504 INFO SenderThread:2123564 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:47,505 INFO SenderThread:2123564 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files |
| 2025-01-06 22:22:47,505 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/config.yaml config.yaml |
| 2025-01-06 22:22:47,505 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 22:22:47,505 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:22:47,508 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/requirements.txt requirements.txt |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log output.log |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/diff.patch diff.patch |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 media/videos/eval/3_eval_3_4cfc3fdd64623711fec5.mp4 |
| 2025-01-06 22:22:47,509 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 media/videos/eval/0_eval_0_51de33a49d1b3553fbba.mp4 |
| 2025-01-06 22:22:47,510 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 media/videos/eval/1_eval_1_5a016504166fce504de7.mp4 |
| 2025-01-06 22:22:47,510 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 media/videos/eval/2_eval_2_4e8e215f602a2441e87a.mp4 |
| 2025-01-06 22:22:47,510 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 media/videos/eval/4_eval_4_c3e93442594f3e5f3d1e.mp4 |
| 2025-01-06 22:22:47,512 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-06 22:22:47,514 INFO SenderThread:2123564 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json media/table/eval/total_success_rate_ci_6_c996a401a7f2ec9cbcab.table.json |
| 2025-01-06 22:22:47,514 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 22:22:47,514 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:22:47,514 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:47,514 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 22:22:47,514 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:47,515 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 22:22:47,515 INFO SenderThread:2123564 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 22:22:47,681 INFO wandb-upload_2:2123564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/config.yaml |
| 2025-01-06 22:22:47,698 INFO wandb-upload_1:2123564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/wandb-summary.json |
| 2025-01-06 22:22:47,735 INFO wandb-upload_3:2123564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/requirements.txt |
| 2025-01-06 22:22:47,742 INFO wandb-upload_0:2123564 [upload_job.py:push():130] Uploaded file /tmp/tmpy2mbo5uowandb/nsvrz20o-events.out.tfevents.1736219890.leonmkim-ROG-Strix-G15CS-G15CS.2123316.0 |
| 2025-01-06 22:22:47,744 INFO wandb-upload_4:2123564 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/files/output.log |
| 2025-01-06 22:22:47,944 INFO Thread-11 (_thread_body):2123564 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 22:22:47,944 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:47,944 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 22:22:47,945 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:47,945 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 22:22:47,945 INFO SenderThread:2123564 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 22:22:47,946 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 22:22:47,946 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:47,946 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 22:22:47,946 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:47,946 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 22:22:47,946 INFO SenderThread:2123564 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 22:22:48,007 INFO SenderThread:2123564 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 22:22:48,007 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 22:22:48,007 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:48,007 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 22:22:48,007 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:48,007 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 22:22:48,007 INFO SenderThread:2123564 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 22:22:48,007 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 22:22:48,007 DEBUG SenderThread:2123564 [sender.py:send():379] send: final |
| 2025-01-06 22:22:48,008 INFO HandlerThread:2123564 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 22:22:48,008 DEBUG SenderThread:2123564 [sender.py:send():379] send: footer |
| 2025-01-06 22:22:48,008 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 22:22:48,008 INFO SenderThread:2123564 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 22:22:48,008 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:22:48,008 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 22:22:48,009 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:22:48,009 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 22:22:48,009 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 22:22:48,009 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 22:22:48,011 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 22:22:48,012 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 22:22:48,061 DEBUG SenderThread:2123564 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 22:22:48,061 INFO MainThread:2123564 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 22:22:48,062 INFO MainThread:2123564 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 22:22:48,062 INFO MainThread:2123564 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 22:22:48,062 DEBUG HandlerThread:2123564 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 22:22:48,062 INFO HandlerThread:2123564 [handler.py:finish():882] shutting down handler |
| 2025-01-06 22:22:49,010 INFO WriterThread:2123564 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/917c47lf/wandb/run-20250106_221810-917c47lf/run-917c47lf.wandb |
| 2025-01-06 22:22:49,061 INFO SenderThread:2123564 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 22:22:49,062 INFO SenderThread:2123564 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 22:22:49,062 INFO SenderThread:2123564 [file_pusher.py:join():175] waiting for file pusher |
|
|
