| 2025-01-06 20:35:57,446 INFO StreamThr :2112019 [internal.py:wandb_internal():85] W&B internal server running at pid: 2112019, started at: 2025-01-06 20:35:57.445835 |
| 2025-01-06 20:35:57,447 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 20:35:57,458 INFO WriterThread:2112019 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/run-vd7nl959.wandb |
| 2025-01-06 20:35:57,459 DEBUG SenderThread:2112019 [sender.py:send():379] send: header |
| 2025-01-06 20:35:57,466 DEBUG SenderThread:2112019 [sender.py:send():379] send: run |
| 2025-01-06 20:35:57,702 INFO SenderThread:2112019 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files |
| 2025-01-06 20:35:57,702 INFO SenderThread:2112019 [sender.py:_start_run_threads():1188] run started: vd7nl959 with start time 1736213757.4465 |
| 2025-01-06 20:35:57,720 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 20:35:57,720 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 20:35:57,805 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 20:35:57,810 DEBUG HandlerThread:2112019 [system_info.py:__init__():26] System info init |
| 2025-01-06 20:35:57,810 DEBUG HandlerThread:2112019 [system_info.py:__init__():41] System info init done |
| 2025-01-06 20:35:57,810 INFO HandlerThread:2112019 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 20:35:57,810 INFO SystemMonitor:2112019 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 20:35:57,810 INFO SystemMonitor:2112019 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 20:35:57,811 INFO HandlerThread:2112019 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 20:35:57,812 INFO SystemMonitor:2112019 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 20:35:57,812 INFO SystemMonitor:2112019 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 20:35:57,813 INFO SystemMonitor:2112019 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 20:35:57,813 INFO SystemMonitor:2112019 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 20:35:57,823 DEBUG HandlerThread:2112019 [system_info.py:probe():152] Probing system |
| 2025-01-06 20:35:57,824 DEBUG HandlerThread:2112019 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 20:35:57,829 DEBUG HandlerThread:2112019 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 20:35:57,829 DEBUG HandlerThread:2112019 [system_info.py:probe():200] Probing system done |
| 2025-01-06 20:35:57,829 DEBUG HandlerThread:2112019 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T01:35:57.823325', 'startedAt': '2025-01-07T01:35:57.437487', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.21275, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.98, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4497.61, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 505.5578384399414}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 20:35:57,829 INFO HandlerThread:2112019 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 20:35:57,829 INFO HandlerThread:2112019 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 20:35:57,829 DEBUG HandlerThread:2112019 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 20:35:57,831 DEBUG HandlerThread:2112019 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 20:35:57,831 DEBUG HandlerThread:2112019 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 20:35:57,915 DEBUG HandlerThread:2112019 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 20:35:57,915 INFO HandlerThread:2112019 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 20:35:57,918 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:35:57,918 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 20:35:57,919 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 20:35:57,919 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 20:35:58,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 20:35:58,159 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 20:35:58,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:35:58,161 DEBUG SenderThread:2112019 [sender.py:send():379] send: telemetry |
| 2025-01-06 20:35:58,181 INFO wandb-upload_2:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/yfojxhab-diff.patch |
| 2025-01-06 20:35:58,181 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/deoj5bdl-wandb-metadata.json |
| 2025-01-06 20:35:58,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:35:58,201 INFO HandlerThread:2112019 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/tb" |
| save: true |
| } |
| control { |
| relay_id: "deba67f5c4a4425e96a5a9a7e46a27bb" |
| } |
| _info { |
| stream_id: "vd7nl959" |
| } |
|
|
| 2025-01-06 20:35:58,201 DEBUG HandlerThread:2112019 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 20:35:58,202 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:35:58,218 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/433434gp-code/FISH/eval_robot.py |
| 2025-01-06 20:35:58,275 DEBUG SenderThread:2112019 [sender.py:send():379] send: tbrecord |
| 2025-01-06 20:35:58,276 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:35:58,276 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 with policy live |
| 2025-01-06 20:35:58,513 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/e2xw3ogu-events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:35:58,704 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:35:58,704 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-metadata.json |
| 2025-01-06 20:35:58,704 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/code/FISH/eval_robot.py |
| 2025-01-06 20:35:58,704 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/diff.patch |
| 2025-01-06 20:35:58,705 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/requirements.txt |
| 2025-01-06 20:35:58,705 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:35:58,705 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/code |
| 2025-01-06 20:35:58,705 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/code/FISH |
| 2025-01-06 20:35:59,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:00,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:00,703 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:01,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:02,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:02,209 DEBUG SenderThread:2112019 [sender.py:send():379] send: config |
| 2025-01-06 20:36:03,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:03,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:04,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:04,704 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:05,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:06,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:07,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:08,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:08,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:08,706 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:09,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:10,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:10,706 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:11,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:12,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:13,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:36:13,159 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:36:13,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:13,295 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:14,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:15,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:16,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:17,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:18,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:19,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:19,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:20,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:21,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:22,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:23,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:24,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:24,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:25,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:26,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:26,710 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:27,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:28,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:36:28,159 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:36:28,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:28,711 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:29,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:29,934 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:30,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:30,712 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/config.yaml |
| 2025-01-06 20:36:30,712 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:31,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:32,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:33,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:34,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:35,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:35,164 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:36,164 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:37,164 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:38,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:39,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:40,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:41,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:41,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:42,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:43,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:36:43,159 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:36:43,205 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:44,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:45,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:46,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:46,310 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:47,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:48,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:49,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:50,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:51,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:52,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:52,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:53,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:54,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:55,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:56,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:56,583 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:36:56,583 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 with policy now |
| 2025-01-06 20:36:56,583 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:36:56,584 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:36:56,584 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:36:56,584 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos |
| 2025-01-06 20:36:56,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval |
| 2025-01-06 20:36:56,848 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/pwh2lllx-media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 |
| 2025-01-06 20:36:57,114 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/be2qygjy-media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 |
| 2025-01-06 20:36:57,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:57,590 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:36:57,813 DEBUG SystemMonitor:2112019 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 20:36:57,814 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:36:58,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:36:58,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:36:58,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:36:58,719 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:36:59,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:00,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:01,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:02,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:02,721 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:03,002 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:03,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:04,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:04,722 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:05,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:05,722 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:37:05,974 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/1czzhzeo-events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:37:06,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:06,722 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:07,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:08,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:08,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:08,723 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:09,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:10,169 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:10,724 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:11,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:12,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:13,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:37:13,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:37:13,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:13,278 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:14,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:15,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:16,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:17,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:18,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:19,162 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:19,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:20,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:21,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:22,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:23,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:24,163 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:24,171 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:25,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:26,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:27,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:27,814 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:37:28,159 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:37:28,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:37:28,201 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:29,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:29,324 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:30,173 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:31,174 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:32,174 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:33,174 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:34,174 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:35,166 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:35,175 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:36,175 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:36,732 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:36,755 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:37:36,755 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 with policy now |
| 2025-01-06 20:37:36,756 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:37:36,768 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:37:36,769 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:37:36,769 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:37:37,005 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/jiyvyyvk-media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 |
| 2025-01-06 20:37:37,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:37,732 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:37:37,732 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 |
| 2025-01-06 20:37:37,733 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval |
| 2025-01-06 20:37:38,024 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/1749qcx4-media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 |
| 2025-01-06 20:37:38,178 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:38,733 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:39,180 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:40,178 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:40,180 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:41,181 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:42,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:42,734 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:43,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:37:43,166 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:37:43,209 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:44,181 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:44,735 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:45,181 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:45,421 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:46,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:47,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:48,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:49,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:50,182 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:51,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:51,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:52,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:53,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:54,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:55,183 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:56,184 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:57,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:37:57,184 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:57,739 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:37:57,815 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:37:58,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:37:58,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:37:58,184 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:37:58,660 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:37:58,660 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:37:58,661 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:37:58,739 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:37:58,740 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:37:59,184 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:00,184 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:00,741 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:01,189 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:02,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:02,579 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:02,741 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:03,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:04,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:04,742 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:05,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:06,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:07,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:07,978 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:08,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:09,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:10,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:11,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:12,185 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:13,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:38:13,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:38:13,205 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:13,251 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:14,186 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:15,186 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:16,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:17,186 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:18,186 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:19,167 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:19,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:20,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:21,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:22,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:23,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:24,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:24,381 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:24,749 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:25,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:26,187 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:27,188 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:27,815 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:38:28,160 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:38:28,160 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:38:28,205 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:29,190 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:29,639 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:29,681 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:38:29,682 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:38:29,682 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 with policy now |
| 2025-01-06 20:38:29,682 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:38:29,682 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:38:29,683 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:38:29,751 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:38:29,751 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 |
| 2025-01-06 20:38:29,751 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval |
| 2025-01-06 20:38:29,944 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/3lx7aqdl-media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 |
| 2025-01-06 20:38:30,194 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:30,214 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/00un4lro-media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 |
| 2025-01-06 20:38:30,751 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:31,197 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:31,751 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:32,202 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:33,210 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:34,204 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:34,752 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:35,203 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:35,541 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:36,204 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:37,207 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:38,207 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:38,754 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:39,208 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:40,208 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:41,136 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:41,206 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:41,452 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:38:41,452 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:38:41,453 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:38:41,755 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:38:41,755 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:38:42,007 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/yyhpk17r-events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:38:42,229 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:42,755 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:43,165 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:38:43,165 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:38:43,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:44,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:44,756 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:45,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:46,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:46,574 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:46,757 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:38:47,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:48,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:49,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:50,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:51,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:52,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:52,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:53,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:54,213 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:55,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:56,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:57,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:57,816 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:38:57,816 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:38:58,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:38:58,161 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:38:58,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:38:59,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:00,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:01,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:02,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:03,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:03,214 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:04,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:05,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:06,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:06,763 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:07,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:08,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:08,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:08,764 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:09,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:10,215 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:11,217 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:12,216 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:13,164 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:39:13,164 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:39:13,221 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:13,324 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:13,629 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:39:13,629 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:39:13,630 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 with policy now |
| 2025-01-06 20:39:13,630 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:39:13,630 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:39:13,630 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:39:13,766 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:39:13,766 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 |
| 2025-01-06 20:39:13,766 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval |
| 2025-01-06 20:39:13,908 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/1vzzwz6d-media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 |
| 2025-01-06 20:39:14,165 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/8pvkcqip-media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 |
| 2025-01-06 20:39:14,243 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:14,766 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:15,224 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:16,224 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:17,237 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:18,237 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:18,768 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:19,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:19,240 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:20,225 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:21,230 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:22,231 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:22,506 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:39:22,506 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:39:22,506 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:39:22,769 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:39:22,769 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:22,769 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:39:23,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:24,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:24,770 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:25,172 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:25,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:26,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:26,770 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:27,231 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:27,816 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:39:28,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:39:28,161 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:39:28,231 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:29,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:30,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:30,276 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:31,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:32,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:33,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:34,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:35,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:36,170 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:36,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:37,232 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:38,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:39,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:40,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:41,174 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:41,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:42,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:42,776 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:43,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:39:43,161 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:39:43,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:44,233 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:45,234 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:46,234 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:46,777 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:47,073 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:47,234 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:48,234 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:49,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:50,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:51,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:52,235 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:52,449 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:52,780 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:53,243 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:53,883 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:39:53,883 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 with policy now |
| 2025-01-06 20:39:53,885 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:39:53,886 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:39:53,886 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:39:53,887 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:39:54,161 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/1dcty9no-media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 |
| 2025-01-06 20:39:54,236 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:54,781 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:54,781 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:39:54,781 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 |
| 2025-01-06 20:39:54,781 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval |
| 2025-01-06 20:39:55,088 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/ew2wcu23-media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 |
| 2025-01-06 20:39:55,248 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:56,245 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:57,244 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:57,718 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:39:57,817 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:39:58,168 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:39:58,168 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:39:58,254 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:39:58,782 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:39:59,266 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:00,257 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:01,254 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:02,254 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:02,555 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:40:02,555 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:02,555 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:02,783 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:02,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:40:02,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:40:02,784 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 20:40:02,784 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 20:40:02,978 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:40:03,029 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/ilxnxfmq-events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:40:03,260 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:03,275 INFO wandb-upload_0:2112019 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpn4nq66mv |
| 2025-01-06 20:40:03,366 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 20:40:03,375 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:40:03,675 INFO SenderThread:2112019 [sender.py:send_request_log_artifact():1525] logged artifact run-vd7nl959-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjMwMjk3Mw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDYzNzE0OQ==', 'latestArtifact': None}} |
| 2025-01-06 20:40:03,675 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:40:03,675 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-06 20:40:03,676 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:40:03,676 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,676 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:40:03,676 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,677 DEBUG SenderThread:2112019 [sender.py:send():379] send: telemetry |
| 2025-01-06 20:40:03,677 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,680 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,680 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,681 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,681 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,681 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,682 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,682 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,682 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,682 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,683 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,683 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,683 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,683 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,684 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,684 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,684 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,684 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,684 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,685 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,685 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:03,685 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:03,685 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 20:40:03,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:03,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:40:03,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json |
| 2025-01-06 20:40:03,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:40:03,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media |
| 2025-01-06 20:40:03,785 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table |
| 2025-01-06 20:40:03,785 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table/eval |
| 2025-01-06 20:40:03,867 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/2ne47f8i-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:40:04,008 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/6bd135am-media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json |
| 2025-01-06 20:40:04,084 INFO wandb-upload_2:2112019 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp4imv0blj |
| 2025-01-06 20:40:04,110 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/5f4vy0vc-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:40:04,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:04,515 INFO SenderThread:2112019 [sender.py:send_request_log_artifact():1525] logged artifact run-vd7nl959-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjMwMjk5Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDYzNzE1MA==', 'latestArtifact': None}} |
| 2025-01-06 20:40:04,515 DEBUG SenderThread:2112019 [sender.py:send():379] send: files |
| 2025-01-06 20:40:04,515 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json with policy now |
| 2025-01-06 20:40:04,516 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:40:04,516 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:04,516 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:04,781 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/rndff0hh-media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json |
| 2025-01-06 20:40:04,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:40:04,784 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:05,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:06,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:07,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:08,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:09,176 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:40:09,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:10,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:11,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:12,255 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:13,161 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:40:13,162 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:40:13,256 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:13,609 DEBUG SenderThread:2112019 [sender.py:send():379] send: exit |
| 2025-01-06 20:40:13,610 INFO SenderThread:2112019 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 20:40:13,610 INFO SenderThread:2112019 [sender.py:send_exit():588] handling runtime: 255 |
| 2025-01-06 20:40:13,611 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:13,611 INFO SenderThread:2112019 [sender.py:send_exit():594] send defer |
| 2025-01-06 20:40:13,611 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:13,611 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 20:40:13,611 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:13,611 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 20:40:13,611 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 20:40:13,611 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:13,611 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 20:40:13,612 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:13,612 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 20:40:13,612 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 20:40:13,612 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:13,612 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 20:40:13,612 INFO HandlerThread:2112019 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 20:40:13,612 DEBUG SystemMonitor:2112019 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 20:40:13,612 DEBUG SystemMonitor:2112019 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 20:40:13,613 INFO HandlerThread:2112019 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 20:40:13,613 INFO HandlerThread:2112019 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 20:40:13,664 INFO HandlerThread:2112019 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 20:40:13,664 INFO HandlerThread:2112019 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 20:40:13,664 INFO HandlerThread:2112019 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 20:40:13,665 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:13,665 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 20:40:13,665 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 20:40:13,665 DEBUG SenderThread:2112019 [sender.py:send():379] send: stats |
| 2025-01-06 20:40:13,665 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:13,665 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 20:40:13,665 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:13,665 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 20:40:13,665 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 20:40:13,666 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:13,666 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 20:40:13,788 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:14,789 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:40:20,562 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:40:20,562 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:40:20,562 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 20:40:20,562 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:40:20,563 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:20,563 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 20:40:20,563 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 20:40:20,563 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:40:20,563 DEBUG SenderThread:2112019 [sender.py:send():379] send: history |
| 2025-01-06 20:40:20,564 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:40:20,564 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:20,564 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 20:40:20,565 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:20,565 DEBUG SenderThread:2112019 [sender.py:send():379] send: summary |
| 2025-01-06 20:40:20,565 INFO SenderThread:2112019 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:40:20,565 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:20,565 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 20:40:20,565 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 20:40:20,565 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:20,566 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 20:40:20,566 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:20,566 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 20:40:20,566 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 20:40:20,566 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:40:20,566 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:20,566 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 20:40:20,566 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:20,566 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 20:40:20,611 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:40:20,791 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/config.yaml |
| 2025-01-06 20:40:20,791 INFO Thread-12 :2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:23,793 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 20:40:23,793 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:40:23,793 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:23,794 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 20:40:23,794 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:23,794 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 20:40:23,794 INFO SenderThread:2112019 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 20:40:23,796 INFO SenderThread:2112019 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 20:40:23,798 INFO SenderThread:2112019 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 20:40:23,813 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 20:40:23,813 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:23,813 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 20:40:23,813 DEBUG SenderThread:2112019 [sender.py:send():379] send: artifact |
| 2025-01-06 20:40:24,101 INFO SenderThread:2112019 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkzNjUwMA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} |
| 2025-01-06 20:40:24,101 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:24,101 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 20:40:24,101 INFO SenderThread:2112019 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 20:40:24,612 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:40:24,792 INFO SenderThread:2112019 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:40:24,792 INFO SenderThread:2112019 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/config.yaml config.yaml |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/requirements.txt requirements.txt |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log output.log |
| 2025-01-06 20:40:24,793 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/diff.patch diff.patch |
| 2025-01-06 20:40:24,795 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 20:40:24,798 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 media/videos/eval/1_eval_1_04da90b434a68619a5e3.mp4 |
| 2025-01-06 20:40:24,798 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 media/videos/eval/2_eval_2_bf3e651ced546db728b0.mp4 |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 media/videos/eval/3_eval_3_9a4bd86fb7d6eb3cf5e4.mp4 |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 media/videos/eval/0_eval_0_1eb31d93c1da1119a0f9.mp4 |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 media/videos/eval/4_eval_4_7007aef1a567b8508ad9.mp4 |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json media/table/eval/total_success_rate_ci_6_633be57ab8f5d2a03a6e.table.json |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:40:24,799 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 20:40:24,799 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:40:24,799 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:24,799 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 20:40:24,801 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:24,801 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 20:40:24,801 INFO SenderThread:2112019 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 20:40:24,988 INFO wandb-upload_2:2112019 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/requirements.txt |
| 2025-01-06 20:40:24,999 INFO wandb-upload_0:2112019 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/wandb-summary.json |
| 2025-01-06 20:40:25,010 INFO wandb-upload_1:2112019 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/config.yaml |
| 2025-01-06 20:40:25,041 INFO wandb-upload_4:2112019 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/files/output.log |
| 2025-01-06 20:40:25,086 INFO wandb-upload_3:2112019 [upload_job.py:push():130] Uploaded file /tmp/tmpvd_mqijuwandb/holtclnn-events.out.tfevents.1736213758.leonmkim-ROG-Strix-G15CS-G15CS.2111854.0 |
| 2025-01-06 20:40:25,286 INFO Thread-11 (_thread_body):2112019 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 20:40:25,286 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:25,286 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 20:40:25,287 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:25,287 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 20:40:25,287 INFO SenderThread:2112019 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 20:40:25,290 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 20:40:25,290 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:25,290 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 20:40:25,290 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:25,290 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 20:40:25,290 INFO SenderThread:2112019 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 20:40:25,361 INFO SenderThread:2112019 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 20:40:25,361 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 20:40:25,361 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:25,361 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 20:40:25,361 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:25,361 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 20:40:25,361 INFO SenderThread:2112019 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 20:40:25,361 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:40:25,361 INFO HandlerThread:2112019 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 20:40:25,362 DEBUG SenderThread:2112019 [sender.py:send():379] send: final |
| 2025-01-06 20:40:25,362 DEBUG SenderThread:2112019 [sender.py:send():379] send: footer |
| 2025-01-06 20:40:25,362 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:40:25,362 INFO SenderThread:2112019 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 20:40:25,362 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:40:25,362 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:40:25,363 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 20:40:25,363 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:40:25,363 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 20:40:25,365 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 20:40:25,365 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 20:40:25,365 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:40:25,412 DEBUG SenderThread:2112019 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:40:25,412 INFO MainThread:2112019 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 20:40:25,413 INFO MainThread:2112019 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 20:40:25,413 INFO MainThread:2112019 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 20:40:25,413 DEBUG HandlerThread:2112019 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 20:40:25,413 INFO HandlerThread:2112019 [handler.py:finish():882] shutting down handler |
| 2025-01-06 20:40:26,363 INFO WriterThread:2112019 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1022_0/vd7nl959/wandb/run-20250106_203557-vd7nl959/run-vd7nl959.wandb |
| 2025-01-06 20:40:26,412 INFO SenderThread:2112019 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 20:40:26,412 INFO SenderThread:2112019 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 20:40:26,412 INFO SenderThread:2112019 [file_pusher.py:join():175] waiting for file pusher |
|
|
