| 2025-01-04 22:12:02,381 INFO StreamThr :2033938 [internal.py:wandb_internal():85] W&B internal server running at pid: 2033938, started at: 2025-01-04 22:12:02.381017 |
| 2025-01-04 22:12:02,382 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status |
| 2025-01-04 22:12:02,394 INFO WriterThread:2033938 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/run-br14p9rb.wandb |
| 2025-01-04 22:12:02,394 DEBUG SenderThread:2033938 [sender.py:send():379] send: header |
| 2025-01-04 22:12:02,405 DEBUG SenderThread:2033938 [sender.py:send():379] send: run |
| 2025-01-04 22:12:02,654 INFO SenderThread:2033938 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files |
| 2025-01-04 22:12:02,654 INFO SenderThread:2033938 [sender.py:_start_run_threads():1188] run started: br14p9rb with start time 1736046722.382562 |
| 2025-01-04 22:12:02,678 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-04 22:12:02,678 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: check_version |
| 2025-01-04 22:12:02,765 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-04 22:12:02,770 DEBUG HandlerThread:2033938 [system_info.py:__init__():26] System info init |
| 2025-01-04 22:12:02,770 DEBUG HandlerThread:2033938 [system_info.py:__init__():41] System info init done |
| 2025-01-04 22:12:02,770 INFO HandlerThread:2033938 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-04 22:12:02,770 INFO SystemMonitor:2033938 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-04 22:12:02,770 INFO HandlerThread:2033938 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-04 22:12:02,771 INFO SystemMonitor:2033938 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-04 22:12:02,771 INFO SystemMonitor:2033938 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-04 22:12:02,772 INFO SystemMonitor:2033938 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-04 22:12:02,772 INFO SystemMonitor:2033938 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-04 22:12:02,773 INFO SystemMonitor:2033938 [interfaces.py:start():188] Started network monitoring |
| 2025-01-04 22:12:02,783 DEBUG HandlerThread:2033938 [system_info.py:probe():152] Probing system |
| 2025-01-04 22:12:02,785 DEBUG HandlerThread:2033938 [system_info.py:_probe_git():137] Probing git |
| 2025-01-04 22:12:02,789 DEBUG HandlerThread:2033938 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-04 22:12:02,789 DEBUG HandlerThread:2033938 [system_info.py:probe():200] Probing system done |
| 2025-01-04 22:12:02,789 DEBUG HandlerThread:2033938 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-05T03:12:02.783618', 'startedAt': '2025-01-05T03:12:02.374537', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '02cf07d901f3d8187a302cd51c9faa80321417ee'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.2435, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.982, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 736.9206237792969}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-04 22:12:02,789 INFO HandlerThread:2033938 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-04 22:12:02,789 INFO HandlerThread:2033938 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-04 22:12:02,789 DEBUG HandlerThread:2033938 [system_info.py:_save_code():44] Saving code |
| 2025-01-04 22:12:02,791 DEBUG HandlerThread:2033938 [system_info.py:_save_code():67] Saving code done |
| 2025-01-04 22:12:02,791 DEBUG HandlerThread:2033938 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-04 22:12:02,853 DEBUG HandlerThread:2033938 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-04 22:12:02,853 INFO HandlerThread:2033938 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-04 22:12:02,856 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:12:02,856 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-04 22:12:02,857 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-04 22:12:02,857 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-04 22:12:03,101 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-04 22:12:03,101 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-04 22:12:03,103 DEBUG SenderThread:2033938 [sender.py:send():379] send: telemetry |
| 2025-01-04 22:12:03,110 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/qq9qy3pd-wandb-metadata.json |
| 2025-01-04 22:12:03,117 INFO wandb-upload_2:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/yxlcxvwt-diff.patch |
| 2025-01-04 22:12:03,131 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/r7lkriq0-code/FISH/eval_robot.py |
| 2025-01-04 22:12:03,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:03,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:12:03,145 INFO HandlerThread:2033938 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/tb" |
| save: true |
| } |
| control { |
| relay_id: "ef5843e7f8e54450bffab8b34651ed96" |
| } |
| _info { |
| stream_id: "br14p9rb" |
| } |
|
|
| 2025-01-04 22:12:03,145 DEBUG HandlerThread:2033938 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-04 22:12:03,146 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:12:03,201 DEBUG SenderThread:2033938 [sender.py:send():379] send: tbrecord |
| 2025-01-04 22:12:03,202 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:12:03,202 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 with policy live |
| 2025-01-04 22:12:03,413 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/bllycg0h-events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/code/FISH/eval_robot.py |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/diff.patch |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-metadata.json |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/requirements.txt |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:12:03,656 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/code/FISH |
| 2025-01-04 22:12:03,657 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/code |
| 2025-01-04 22:12:04,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:05,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:05,655 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:06,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:07,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:08,063 DEBUG SenderThread:2033938 [sender.py:send():379] send: config |
| 2025-01-04 22:12:08,063 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:08,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:09,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:09,657 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:10,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:11,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:11,658 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:12,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:13,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:13,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:13,659 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:14,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:15,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:15,660 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:16,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:17,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:17,660 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:18,101 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:12:18,102 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:12:18,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:18,168 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:19,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:20,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:21,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:22,104 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:23,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:24,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:24,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:25,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:26,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:27,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:27,663 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:28,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:29,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:29,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:29,664 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:30,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:31,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:31,664 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:32,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:33,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:12:33,106 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:12:33,149 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:34,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:34,261 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:34,665 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/config.yaml |
| 2025-01-04 22:12:35,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:36,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:37,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:38,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:39,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:40,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:40,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:41,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:41,667 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:42,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:43,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:44,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:45,105 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:45,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:45,668 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:46,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:47,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:47,669 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:12:48,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:12:48,102 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:12:48,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:49,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:50,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:50,248 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:51,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:52,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:53,109 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:54,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:55,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:56,106 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:12:56,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:57,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:58,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:12:59,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:00,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:01,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:01,269 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:01,672 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:02,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:02,773 DEBUG SystemMonitor:2033938 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-04 22:13:02,774 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:13:03,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:13:03,102 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:13:03,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:03,673 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:04,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:05,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:05,854 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:13:05,854 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 with policy now |
| 2025-01-04 22:13:05,854 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:13:05,855 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:13:05,855 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:13:05,856 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:13:06,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:06,140 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/3gzygjhq-media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 |
| 2025-01-04 22:13:06,674 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:13:06,674 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 |
| 2025-01-04 22:13:06,674 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media |
| 2025-01-04 22:13:06,675 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval |
| 2025-01-04 22:13:06,675 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos |
| 2025-01-04 22:13:06,864 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:06,938 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/fem9d0y3-media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 |
| 2025-01-04 22:13:07,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:07,674 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:08,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:09,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:09,675 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:10,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:11,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:11,676 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:12,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:12,521 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:13,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:13,677 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:14,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:15,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:16,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:17,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:18,102 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:13:18,102 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:13:18,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:18,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:19,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:20,112 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:21,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:22,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:23,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:23,358 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:23,680 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:13:23,909 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/lor7etkm-events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:13:24,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:25,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:25,681 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:26,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:27,121 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:28,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:29,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:29,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:29,682 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:30,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:31,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:32,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:32,777 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:13:33,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:13:33,103 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:13:33,145 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:34,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:34,186 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:35,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:36,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:37,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:38,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:39,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:40,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:40,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:41,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:42,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:43,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:44,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:45,107 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:45,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:46,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:47,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:48,103 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:13:48,103 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:13:48,149 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:49,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:50,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:50,247 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:51,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:52,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:53,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:53,690 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:13:54,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:55,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:56,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:13:56,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:57,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:58,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:13:59,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:00,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:01,108 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:01,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:02,119 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:02,775 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:14:03,110 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:14:03,111 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:14:03,124 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:03,301 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:14:03,301 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:14:03,302 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 with policy now |
| 2025-01-04 22:14:03,302 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:14:03,302 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:14:03,328 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:14:03,560 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/fznmglo3-media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 |
| 2025-01-04 22:14:03,693 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:14:03,693 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:03,694 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 |
| 2025-01-04 22:14:03,694 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval |
| 2025-01-04 22:14:03,946 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/no3h86jb-media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 |
| 2025-01-04 22:14:04,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:05,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:05,694 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:06,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:06,329 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:07,136 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:07,695 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:08,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:09,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:10,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:11,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:11,505 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:11,696 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:12,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:13,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:13,697 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:14:14,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:14,572 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:14:14,572 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:14:14,573 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:14:14,697 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:14:15,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:15,697 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:16,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:16,661 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:17,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:18,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:14:18,111 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:14:18,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:19,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:19,699 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:20,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:21,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:22,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:22,260 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:23,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:24,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:25,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:26,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:27,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:28,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:28,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:29,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:30,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:31,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:32,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:32,775 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:14:33,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:14:33,111 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:14:33,153 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:33,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:34,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:35,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:36,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:37,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:38,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:39,114 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:39,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:40,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:41,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:42,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:43,134 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:44,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:44,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:45,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:46,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:47,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:48,111 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:14:48,111 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:14:48,153 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:49,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:49,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:50,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:51,135 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:52,136 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:53,140 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:53,281 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:14:53,281 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:14:53,282 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 with policy now |
| 2025-01-04 22:14:53,282 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:14:53,282 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:14:53,285 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:14:53,534 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/1sg2i1ej-media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 |
| 2025-01-04 22:14:53,710 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:14:53,710 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:53,710 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 |
| 2025-01-04 22:14:53,710 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval |
| 2025-01-04 22:14:53,968 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/24etsvzs-media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 |
| 2025-01-04 22:14:54,142 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:54,308 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:14:55,147 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:55,710 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:56,147 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:57,149 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:57,711 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:14:58,150 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:59,152 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:14:59,406 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:00,148 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:01,153 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:01,713 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:15:02,146 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:02,776 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:15:03,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:15:03,114 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:15:03,157 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:04,151 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:05,152 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:05,193 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:06,151 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:07,153 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:08,149 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:09,151 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:10,152 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:11,123 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:11,151 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:12,158 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:13,155 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:14,156 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:15,158 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:16,157 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:17,122 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:17,155 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:18,113 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:15:18,114 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:15:18,161 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:19,158 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:20,155 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:21,155 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:22,166 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:22,205 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:23,158 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:24,160 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:25,161 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:26,162 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:27,161 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:28,121 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:28,161 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:29,161 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:30,160 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:31,164 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:32,165 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:32,776 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:15:33,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:15:33,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:15:33,168 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:33,174 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:34,165 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:35,167 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:36,165 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:37,166 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:38,168 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:39,122 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:39,172 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:40,169 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:41,174 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:42,179 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:43,175 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:44,177 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:45,120 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:45,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:46,178 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:47,176 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:47,461 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:15:47,462 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:15:47,463 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:15:47,728 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:15:47,729 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:15:47,966 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/95u33p4l-events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:15:48,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:15:48,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:15:48,178 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:49,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:49,729 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:15:50,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:50,501 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:51,185 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:52,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:53,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:53,730 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:15:54,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:55,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:56,120 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:15:56,181 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:57,180 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:58,181 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:15:59,181 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:00,181 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:01,181 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:01,389 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:01,733 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:02,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:02,776 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:16:03,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:16:03,116 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:16:03,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:04,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:05,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:05,735 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:06,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:07,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:07,285 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:08,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:09,182 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:10,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:11,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:11,737 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:12,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:12,505 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:13,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:14,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:15,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:15,739 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:16,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:17,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:17,657 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:18,115 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:16:18,115 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:16:18,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:19,183 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:20,184 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:21,184 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:22,184 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:23,121 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:23,184 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:24,185 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:25,185 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:26,185 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:27,187 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:27,365 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:16:27,365 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:16:27,365 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 with policy now |
| 2025-01-04 22:16:27,366 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:16:27,366 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:16:27,366 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:16:27,625 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/htmi0rq3-media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 |
| 2025-01-04 22:16:27,743 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:16:27,743 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:27,743 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 |
| 2025-01-04 22:16:27,743 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval |
| 2025-01-04 22:16:28,016 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/fnli2a9z-media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 |
| 2025-01-04 22:16:28,123 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:28,190 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:29,186 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:29,744 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:30,187 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:31,190 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:31,744 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:32,187 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:32,777 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:16:33,120 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:16:33,120 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:16:33,189 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:33,253 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:34,194 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:35,195 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:35,746 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:16:36,200 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:37,198 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:38,200 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:39,124 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:39,198 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:40,200 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:41,203 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:42,204 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:43,205 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:44,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:44,202 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:45,202 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:46,202 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:47,206 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:48,119 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:16:48,120 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:16:48,204 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:49,209 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:49,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:50,206 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:51,205 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:52,208 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:53,211 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:54,208 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:55,126 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:16:55,209 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:56,208 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:57,209 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:58,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:58,560 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:16:58,560 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:16:58,561 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:16:58,755 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:16:58,755 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:16:58,982 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/9scxdb02-events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:16:59,220 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:16:59,755 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:00,218 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:00,941 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:01,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:02,211 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:02,777 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:17:03,116 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:17:03,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:17:03,211 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:03,757 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:04,211 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:05,211 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:06,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:06,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:07,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:08,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:09,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:09,759 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:10,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:11,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:12,125 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:12,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:13,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:14,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:15,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:16,212 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:17,125 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:17,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:18,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:17:18,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:17:18,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:19,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:20,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:21,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:22,198 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:22,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:23,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:24,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:25,213 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:25,764 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:26,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:27,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:28,125 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:28,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:29,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:29,766 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:30,214 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:31,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:32,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:32,780 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:17:33,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:17:33,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:17:33,209 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:33,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:34,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:35,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:36,215 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:37,221 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:37,770 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:37,912 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:17:37,912 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 with policy now |
| 2025-01-04 22:17:37,918 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:17:37,919 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:17:37,919 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:37,922 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:38,175 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/l3qx37z0-media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 |
| 2025-01-04 22:17:38,217 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:38,770 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:17:38,770 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 |
| 2025-01-04 22:17:38,771 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval |
| 2025-01-04 22:17:39,063 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/0z6dseik-media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 |
| 2025-01-04 22:17:39,125 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:39,221 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:39,771 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:40,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:41,224 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:42,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:43,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:43,772 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:44,138 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:44,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:45,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:45,773 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:46,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:46,618 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:17:46,618 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:46,619 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:46,773 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:17:46,773 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:17:47,026 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-04 22:17:47,026 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-04 22:17:47,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:47,306 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:17:47,449 INFO wandb-upload_0:2033938 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpk9l0lb5b |
| 2025-01-04 22:17:47,706 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-04 22:17:47,718 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 22:17:47,773 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table |
| 2025-01-04 22:17:47,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table/eval |
| 2025-01-04 22:17:47,794 INFO SenderThread:2033938 [sender.py:send_request_log_artifact():1525] logged artifact run-br14p9rb-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTE4NTYwOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg2MTkwOQ==', 'latestArtifact': None}} |
| 2025-01-04 22:17:47,794 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:17:47,794 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json with policy now |
| 2025-01-04 22:17:47,795 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:17:47,795 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,798 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,798 DEBUG SenderThread:2033938 [sender.py:send():379] send: telemetry |
| 2025-01-04 22:17:47,798 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,799 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,799 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,799 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,799 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,800 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,800 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,801 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,801 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,801 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,801 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,802 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,802 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,802 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,802 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,802 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,803 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,803 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,803 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,803 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,803 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:47,804 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:47,804 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-04 22:17:47,978 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/tdeyyxvo-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-04 22:17:47,981 INFO wandb-upload_2:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/kqlpoxzf-media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json |
| 2025-01-04 22:17:48,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:17:48,167 INFO wandb-upload_0:2033938 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpol9ft8ro |
| 2025-01-04 22:17:48,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:48,306 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/jpbnk4yl-media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-04 22:17:48,528 INFO SenderThread:2033938 [sender.py:send_request_log_artifact():1525] logged artifact run-br14p9rb-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTE4NTYyNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg2MTkxNA==', 'latestArtifact': None}} |
| 2025-01-04 22:17:48,528 DEBUG SenderThread:2033938 [sender.py:send():379] send: files |
| 2025-01-04 22:17:48,528 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json with policy now |
| 2025-01-04 22:17:48,528 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:17:48,529 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:17:48,529 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:17:48,529 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:17:48,759 INFO wandb-upload_2:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/xgtg4wb1-media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json |
| 2025-01-04 22:17:48,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:17:49,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:49,666 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:49,774 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:17:50,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:51,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:52,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:53,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:54,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:55,129 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:17:55,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:56,225 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:57,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:58,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:17:59,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:00,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:00,247 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:00,779 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/config.yaml |
| 2025-01-04 22:18:01,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:02,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:02,778 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:18:03,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:18:03,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:18:03,226 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:04,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:05,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:05,276 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:06,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:07,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:08,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:09,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:10,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:11,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:11,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:12,227 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:13,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:14,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:15,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:16,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:16,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:17,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:18,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:18:18,117 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:18:18,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:19,228 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:20,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:21,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:21,275 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:22,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:23,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:24,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:25,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:26,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:27,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:27,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:28,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:29,229 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:30,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:31,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:32,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:32,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:32,779 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:18:33,117 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:18:33,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:18:33,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:34,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:35,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:36,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:37,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:37,263 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:38,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:39,230 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:40,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:41,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:42,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:43,130 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:43,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:44,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:45,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:46,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:47,231 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:48,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:18:48,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:18:48,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:48,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:49,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:50,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:51,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:52,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:53,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:54,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:54,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:55,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:56,232 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:57,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:58,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:18:59,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:18:59,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:00,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:01,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:02,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:02,779 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:19:03,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:19:03,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:19:03,233 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:04,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:04,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:05,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:06,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:07,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:08,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:09,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:10,131 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:10,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:11,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:12,234 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:13,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:14,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:15,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:15,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:16,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:17,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:18,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:19:18,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:19:18,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:19,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:20,235 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:20,237 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:21,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:22,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:23,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:24,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:25,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:26,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:26,236 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:27,237 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:28,237 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:29,237 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:30,237 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:31,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:32,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:32,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:32,780 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:19:33,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:19:33,118 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:19:33,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:34,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:35,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:36,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:37,238 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:37,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:38,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:39,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:40,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:41,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:42,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:43,132 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:43,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:44,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:45,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:46,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:47,239 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:48,118 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:19:48,119 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:19:48,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:48,249 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:49,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:50,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:51,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:52,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:53,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:54,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:54,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:55,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:56,240 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:57,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:58,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:19:59,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:19:59,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:00,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:01,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:02,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:02,780 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:20:03,119 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 22:20:03,119 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 22:20:03,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:04,202 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:20:04,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:05,241 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:06,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:07,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:08,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:09,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:10,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:20:10,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:11,242 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:12,243 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:13,243 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:14,243 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:15,133 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:20:15,243 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:15,329 DEBUG SenderThread:2033938 [sender.py:send():379] send: exit |
| 2025-01-04 22:20:15,329 INFO SenderThread:2033938 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-04 22:20:15,330 INFO SenderThread:2033938 [sender.py:send_exit():588] handling runtime: 492 |
| 2025-01-04 22:20:15,330 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:20:15,330 INFO SenderThread:2033938 [sender.py:send_exit():594] send defer |
| 2025-01-04 22:20:15,330 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:15,330 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-04 22:20:15,330 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:15,330 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-04 22:20:15,330 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-04 22:20:15,330 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:15,331 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-04 22:20:15,331 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:15,331 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-04 22:20:15,331 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-04 22:20:15,331 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:15,331 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-04 22:20:15,331 INFO HandlerThread:2033938 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-04 22:20:15,331 DEBUG SystemMonitor:2033938 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-04 22:20:15,331 DEBUG SystemMonitor:2033938 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-04 22:20:15,332 INFO HandlerThread:2033938 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-04 22:20:15,332 INFO HandlerThread:2033938 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-04 22:20:15,383 INFO HandlerThread:2033938 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-04 22:20:15,384 INFO HandlerThread:2033938 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-04 22:20:15,384 INFO HandlerThread:2033938 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-04 22:20:15,384 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:15,384 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-04 22:20:15,384 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-04 22:20:15,384 DEBUG SenderThread:2033938 [sender.py:send():379] send: stats |
| 2025-01-04 22:20:15,385 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:15,385 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-04 22:20:15,385 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:15,385 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-04 22:20:15,385 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-04 22:20:15,385 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:15,385 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-04 22:20:15,835 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:20:17,836 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:20:21,644 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 22:20:21,644 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:20:21,644 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-04 22:20:21,645 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,645 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-04 22:20:21,645 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-04 22:20:21,645 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 22:20:21,646 DEBUG SenderThread:2033938 [sender.py:send():379] send: history |
| 2025-01-04 22:20:21,646 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 22:20:21,646 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:20:21,646 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:21,646 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-04 22:20:21,647 DEBUG SenderThread:2033938 [sender.py:send():379] send: summary |
| 2025-01-04 22:20:21,647 INFO SenderThread:2033938 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 22:20:21,647 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,647 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-04 22:20:21,647 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-04 22:20:21,648 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:21,648 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-04 22:20:21,648 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,648 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-04 22:20:21,648 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-04 22:20:21,648 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 22:20:21,648 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:21,649 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-04 22:20:21,649 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,649 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-04 22:20:21,753 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-04 22:20:21,753 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:21,753 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-04 22:20:21,753 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,753 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-04 22:20:21,754 INFO SenderThread:2033938 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-04 22:20:21,756 INFO SenderThread:2033938 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-04 22:20:21,758 INFO SenderThread:2033938 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-04 22:20:21,769 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-04 22:20:21,769 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:21,769 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-04 22:20:21,769 DEBUG SenderThread:2033938 [sender.py:send():379] send: artifact |
| 2025-01-04 22:20:21,838 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:20:21,838 INFO Thread-12 :2033938 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:20:21,938 INFO SenderThread:2033938 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxOTEwMTI1NQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxOTEzMDY5OQ==', 'versionIndex': 419}}} |
| 2025-01-04 22:20:21,938 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:21,938 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-04 22:20:21,938 INFO SenderThread:2033938 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-04 22:20:22,330 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/config.yaml config.yaml |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json wandb-summary.json |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/requirements.txt requirements.txt |
| 2025-01-04 22:20:22,839 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-04 22:20:22,840 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log output.log |
| 2025-01-04 22:20:22,843 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/diff.patch diff.patch |
| 2025-01-04 22:20:22,845 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-04 22:20:22,848 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 media/videos/eval/2_eval_2_10e1974516cae1f65fc3.mp4 |
| 2025-01-04 22:20:22,848 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 media/videos/eval/3_eval_3_129e9d1f1ab8c01c94d0.mp4 |
| 2025-01-04 22:20:22,848 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 media/videos/eval/4_eval_4_4792f0829cf66c6b9d8a.mp4 |
| 2025-01-04 22:20:22,848 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 media/videos/eval/0_eval_0_92869d54d5ca1018f4a9.mp4 |
| 2025-01-04 22:20:22,848 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 media/videos/eval/1_eval_1_363a77cf1a9e04df3395.mp4 |
| 2025-01-04 22:20:22,849 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json media/table/eval/total_success_rate_ci_6_f6c18c2ba742daffddcb.table.json |
| 2025-01-04 22:20:22,849 INFO SenderThread:2033938 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json media/table/eval/success_rate_ci_5_ff553b4df26f1e2b1b21.table.json |
| 2025-01-04 22:20:22,849 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-04 22:20:22,849 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 22:20:22,849 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:22,849 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-04 22:20:22,849 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:22,849 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-04 22:20:22,849 INFO SenderThread:2033938 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-04 22:20:23,011 INFO wandb-upload_0:2033938 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/wandb-summary.json |
| 2025-01-04 22:20:23,030 INFO wandb-upload_1:2033938 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/config.yaml |
| 2025-01-04 22:20:23,083 INFO wandb-upload_3:2033938 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/requirements.txt |
| 2025-01-04 22:20:23,083 INFO wandb-upload_2:2033938 [upload_job.py:push():130] Uploaded file /tmp/tmpv289gai6wandb/qt0vmmye-events.out.tfevents.1736046723.leonmkim-ROG-Strix-G15CS-G15CS.2033763.0 |
| 2025-01-04 22:20:23,104 INFO wandb-upload_4:2033938 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/files/output.log |
| 2025-01-04 22:20:23,304 INFO Thread-11 (_thread_body):2033938 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-04 22:20:23,304 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:23,304 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-04 22:20:23,305 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:23,305 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-04 22:20:23,305 INFO SenderThread:2033938 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-04 22:20:23,306 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-04 22:20:23,306 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:23,306 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-04 22:20:23,306 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:23,306 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-04 22:20:23,306 INFO SenderThread:2033938 [file_stream.py:finish():601] file stream finish called |
| 2025-01-04 22:20:23,331 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 22:20:23,578 INFO SenderThread:2033938 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-04 22:20:23,578 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-04 22:20:23,578 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 22:20:23,578 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:23,578 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-04 22:20:23,579 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:23,579 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-04 22:20:23,579 INFO SenderThread:2033938 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-04 22:20:23,579 DEBUG SenderThread:2033938 [sender.py:send():379] send: final |
| 2025-01-04 22:20:23,579 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 22:20:23,579 DEBUG SenderThread:2033938 [sender.py:send():379] send: footer |
| 2025-01-04 22:20:23,579 INFO HandlerThread:2033938 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-04 22:20:23,579 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 22:20:23,579 INFO SenderThread:2033938 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-04 22:20:23,580 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 22:20:23,580 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-04 22:20:23,580 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 22:20:23,580 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-04 22:20:23,580 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-04 22:20:23,581 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 22:20:23,581 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 22:20:23,581 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: server_info |
| 2025-01-04 22:20:23,646 DEBUG SenderThread:2033938 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 22:20:23,646 INFO MainThread:2033938 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-04 22:20:23,646 INFO MainThread:2033938 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-04 22:20:23,647 INFO MainThread:2033938 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-04 22:20:23,647 DEBUG HandlerThread:2033938 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-04 22:20:23,647 INFO HandlerThread:2033938 [handler.py:finish():882] shutting down handler |
| 2025-01-04 22:20:24,581 INFO WriterThread:2033938 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/br14p9rb/wandb/run-20250104_221202-br14p9rb/run-br14p9rb.wandb |
| 2025-01-04 22:20:24,646 INFO SenderThread:2033938 [sender.py:finish():1615] shutting down sender |
| 2025-01-04 22:20:24,646 INFO SenderThread:2033938 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-04 22:20:24,646 INFO SenderThread:2033938 [file_pusher.py:join():175] waiting for file pusher |
|
|
