| 2025-01-21 21:06:51,388 INFO StreamThr :2645977 [internal.py:wandb_internal():85] W&B internal server running at pid: 2645977, started at: 2025-01-21 21:06:51.388319 |
| 2025-01-21 21:06:51,389 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status |
| 2025-01-21 21:06:51,397 INFO WriterThread:2645977 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/run-bn7ul4i2.wandb |
| 2025-01-21 21:06:51,398 DEBUG SenderThread:2645977 [sender.py:send():379] send: header |
| 2025-01-21 21:06:51,407 DEBUG SenderThread:2645977 [sender.py:send():379] send: run |
| 2025-01-21 21:06:51,716 INFO SenderThread:2645977 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files |
| 2025-01-21 21:06:51,717 INFO SenderThread:2645977 [sender.py:_start_run_threads():1188] run started: bn7ul4i2 with start time 1737511611.388322 |
| 2025-01-21 21:06:51,735 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-21 21:06:51,735 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: check_version |
| 2025-01-21 21:06:51,781 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-21 21:06:51,785 DEBUG HandlerThread:2645977 [system_info.py:__init__():26] System info init |
| 2025-01-21 21:06:51,786 DEBUG HandlerThread:2645977 [system_info.py:__init__():41] System info init done |
| 2025-01-21 21:06:51,786 INFO HandlerThread:2645977 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-21 21:06:51,786 INFO SystemMonitor:2645977 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-21 21:06:51,786 INFO HandlerThread:2645977 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-21 21:06:51,787 INFO SystemMonitor:2645977 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-21 21:06:51,787 INFO SystemMonitor:2645977 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-21 21:06:51,787 INFO SystemMonitor:2645977 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-21 21:06:51,788 INFO SystemMonitor:2645977 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-21 21:06:51,788 INFO SystemMonitor:2645977 [interfaces.py:start():188] Started network monitoring |
| 2025-01-21 21:06:51,799 DEBUG HandlerThread:2645977 [system_info.py:probe():152] Probing system |
| 2025-01-21 21:06:51,801 DEBUG HandlerThread:2645977 [system_info.py:_probe_git():137] Probing git |
| 2025-01-21 21:06:51,805 DEBUG HandlerThread:2645977 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-21 21:06:51,805 DEBUG HandlerThread:2645977 [system_info.py:probe():200] Probing system done |
| 2025-01-21 21:06:51,805 DEBUG HandlerThread:2645977 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-22T02:06:51.799700', 'startedAt': '2025-01-22T02:06:51.381170', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'dc9da9fa0ef2996c7fec8d105e7933e728cd952f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.5710000000004, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4497.405, 'min': 800.0, 'max': 4700.0}, {'current': 4499.988, 'min': 800.0, 'max': 4700.0}, {'current': 4496.469, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.983, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.951, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 344.01396560668945}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-21 21:06:51,806 INFO HandlerThread:2645977 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-21 21:06:51,806 INFO HandlerThread:2645977 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-21 21:06:51,806 DEBUG HandlerThread:2645977 [system_info.py:_save_code():44] Saving code |
| 2025-01-21 21:06:51,807 DEBUG HandlerThread:2645977 [system_info.py:_save_code():67] Saving code done |
| 2025-01-21 21:06:51,807 DEBUG HandlerThread:2645977 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-21 21:06:51,869 DEBUG HandlerThread:2645977 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-21 21:06:51,869 INFO HandlerThread:2645977 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-21 21:06:51,872 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:06:51,872 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-21 21:06:51,872 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-21 21:06:51,873 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-21 21:06:52,114 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-21 21:06:52,114 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-21 21:06:52,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:06:52,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:06:52,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:52,117 INFO HandlerThread:2645977 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/tb" |
| save: true |
| } |
| control { |
| relay_id: "8fcde4342c68447694e117753e76da0a" |
| } |
| _info { |
| stream_id: "bn7ul4i2" |
| } |
|
|
| 2025-01-21 21:06:52,117 DEBUG HandlerThread:2645977 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-21 21:06:52,125 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/u8tdtv45-wandb-metadata.json |
| 2025-01-21 21:06:52,143 INFO wandb-upload_1:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/3pw2oj2i-code/FISH/eval_robot.py |
| 2025-01-21 21:06:52,180 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/dukk9114-diff.patch |
| 2025-01-21 21:06:52,185 DEBUG SenderThread:2645977 [sender.py:send():379] send: telemetry |
| 2025-01-21 21:06:52,185 DEBUG SenderThread:2645977 [sender.py:send():379] send: tbrecord |
| 2025-01-21 21:06:52,186 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:06:52,186 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 with policy live |
| 2025-01-21 21:06:52,388 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/8dovbc8h-events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:06:52,718 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-metadata.json |
| 2025-01-21 21:06:52,718 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:06:52,718 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:06:52,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/code/FISH/eval_robot.py |
| 2025-01-21 21:06:52,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/diff.patch |
| 2025-01-21 21:06:52,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/requirements.txt |
| 2025-01-21 21:06:52,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/code/FISH |
| 2025-01-21 21:06:52,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/code |
| 2025-01-21 21:06:53,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:54,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:54,717 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:06:55,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:56,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:56,540 DEBUG SenderThread:2645977 [sender.py:send():379] send: config |
| 2025-01-21 21:06:56,540 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:06:57,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:58,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:06:58,719 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:06:59,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:00,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:01,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:02,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:02,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:02,720 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:03,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:04,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:05,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:06,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:06,721 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:07,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:07:07,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:07:07,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:07,277 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:08,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:08,721 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:09,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:10,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:10,722 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:11,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:12,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:12,723 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:12,913 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:13,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:14,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:14,723 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:15,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:16,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:17,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:18,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:18,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:19,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:20,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:21,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:22,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:07:22,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:07:22,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:23,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:23,701 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:24,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:24,726 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/config.yaml |
| 2025-01-21 21:07:25,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:26,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:27,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:28,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:29,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:29,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:30,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:31,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:32,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:33,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:34,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:35,116 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:35,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:36,121 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:37,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:07:37,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:07:37,121 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:37,405 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:07:37,406 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:07:37,406 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 with policy now |
| 2025-01-21 21:07:37,406 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:07:37,406 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:07:37,407 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:07:37,698 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/7seopqdc-media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 |
| 2025-01-21 21:07:37,729 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 |
| 2025-01-21 21:07:37,729 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:07:37,730 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos |
| 2025-01-21 21:07:37,730 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval |
| 2025-01-21 21:07:37,730 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media |
| 2025-01-21 21:07:38,008 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/4cu6sdrr-media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 |
| 2025-01-21 21:07:38,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:38,730 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:39,121 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:40,121 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:40,408 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:41,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:42,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:42,731 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:43,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:44,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:44,732 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:45,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:45,737 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:46,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:47,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:48,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:49,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:49,733 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:07:50,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:50,734 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:51,037 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:51,122 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:51,788 DEBUG SystemMonitor:2645977 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-21 21:07:51,789 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:07:52,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:07:52,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:07:52,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:52,735 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:53,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:54,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:54,735 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:07:55,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:56,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:56,948 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:07:57,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:58,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:07:59,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:00,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:01,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:02,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:02,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:03,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:04,124 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:05,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:06,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:07,115 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:08:07,115 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:08:07,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:07,256 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:08,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:09,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:10,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:11,125 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:12,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:13,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:13,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:14,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:15,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:16,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:17,126 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:18,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:18,127 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:19,127 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:20,134 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:20,743 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:20,974 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:08:20,974 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 with policy now |
| 2025-01-21 21:08:20,974 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:08:20,975 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:08:20,975 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:08:20,975 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:08:21,129 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:21,263 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/gvmll21x-media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 |
| 2025-01-21 21:08:21,744 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:08:21,744 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 |
| 2025-01-21 21:08:21,744 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval |
| 2025-01-21 21:08:21,789 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:08:22,068 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/g0sx2ln3-media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 |
| 2025-01-21 21:08:22,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:08:22,120 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:08:22,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:22,744 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:23,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:23,277 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:23,744 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:24,134 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:25,132 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:26,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:26,745 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:27,139 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:28,132 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:28,417 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:28,746 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:29,132 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:30,132 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:31,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:32,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:33,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:34,074 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:34,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:34,592 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:08:34,592 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:08:34,593 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:08:34,748 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:08:34,749 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:08:35,038 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/287aceph-events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:08:35,149 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:36,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:36,749 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:37,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:08:37,120 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:08:37,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:38,134 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:38,750 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:39,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:39,476 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:40,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:40,751 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:41,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:42,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:43,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:44,135 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:45,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:45,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:46,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:47,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:48,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:49,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:50,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:50,921 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:51,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:51,790 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:08:52,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:08:52,117 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:08:52,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:52,755 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:53,136 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:54,149 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:55,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:56,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:56,150 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:08:56,756 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:08:57,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:58,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:08:59,149 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:00,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:00,757 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:01,138 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:02,121 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:02,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:03,138 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:04,138 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:04,759 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:05,139 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:06,141 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:06,265 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:09:06,266 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:09:06,266 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 with policy now |
| 2025-01-21 21:09:06,266 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:09:06,267 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:09:06,267 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:09:06,550 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/w49ytuch-media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 |
| 2025-01-21 21:09:06,760 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:06,760 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:09:06,760 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 |
| 2025-01-21 21:09:06,760 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval |
| 2025-01-21 21:09:07,102 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/3862svq1-media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 |
| 2025-01-21 21:09:07,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:09:07,120 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:09:07,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:07,257 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:08,155 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:08,760 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:09,143 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:10,141 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:11,143 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:12,143 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:12,731 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:12,762 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:13,142 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:14,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:14,764 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:15,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:16,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:17,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:17,765 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:09:18,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:18,352 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:09:18,352 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:09:18,353 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:18,353 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:09:18,765 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:18,765 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:09:19,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:20,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:20,766 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:21,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:21,791 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:09:22,117 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:09:22,117 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:09:22,144 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:22,766 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:23,144 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:24,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:24,261 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:25,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:26,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:27,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:28,145 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:29,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:30,123 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:30,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:31,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:32,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:33,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:34,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:35,127 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:35,146 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:36,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:37,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:09:37,118 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:09:37,161 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:38,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:39,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:40,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:40,247 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:41,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:42,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:43,147 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:44,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:45,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:46,129 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:46,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:47,148 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:48,149 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:48,776 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:49,152 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:50,033 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:09:50,033 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 with policy now |
| 2025-01-21 21:09:50,036 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:09:50,037 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:09:50,037 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:09:50,037 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:09:50,152 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:50,341 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/fwhbx8yt-media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 |
| 2025-01-21 21:09:50,777 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:50,777 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:09:50,777 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 |
| 2025-01-21 21:09:50,778 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval |
| 2025-01-21 21:09:51,093 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/wpl0pvln-media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 |
| 2025-01-21 21:09:51,159 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:51,777 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:51,791 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:09:51,792 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:52,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:09:52,133 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:09:52,177 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:52,778 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:53,164 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:54,163 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:54,778 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:55,172 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:56,166 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:57,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:09:57,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:58,178 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:09:58,780 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:09:59,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:00,172 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:01,180 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:02,138 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:02,172 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:02,407 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:10:02,407 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:02,408 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:02,781 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:10:02,781 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:10:03,062 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/y8fiahmw-events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:10:03,174 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:04,178 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:04,782 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:05,181 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:06,172 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:06,783 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:07,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:10:07,119 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:10:07,172 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:07,204 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:08,173 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:08,783 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:09,173 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:10,173 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:11,173 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:12,173 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:12,785 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:13,131 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:13,174 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:14,174 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:15,174 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:16,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:17,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:18,131 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:18,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:19,174 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:20,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:21,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:21,792 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:10:22,118 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:10:22,119 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:10:22,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:23,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:23,272 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:24,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:25,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:26,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:27,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:28,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:29,131 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:29,175 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:30,176 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:31,176 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:32,176 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:33,176 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:34,181 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:34,183 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:34,229 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:10:34,229 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 with policy now |
| 2025-01-21 21:10:34,230 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:10:34,230 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:10:34,230 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:34,231 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:34,517 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/7ygne6de-media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 |
| 2025-01-21 21:10:34,792 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:34,793 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:10:34,793 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 |
| 2025-01-21 21:10:34,793 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval |
| 2025-01-21 21:10:35,086 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/kzqyng5g-media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 |
| 2025-01-21 21:10:35,185 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:35,793 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:36,181 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:36,793 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:37,120 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:10:37,120 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:10:37,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:38,189 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:38,794 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:39,190 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:39,913 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:40,191 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:41,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:42,199 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:42,795 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:43,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:43,456 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:10:43,456 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:43,457 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:43,609 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 21:10:43,609 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval |
| 2025-01-21 21:10:43,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table |
| 2025-01-21 21:10:43,843 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:10:44,000 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/4f5g6jxq-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-21 21:10:44,134 INFO wandb-upload_0:2645977 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpmp6_2_p8 |
| 2025-01-21 21:10:44,192 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:44,246 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 21:10:44,289 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:10:44,553 INFO SenderThread:2645977 [sender.py:send_request_log_artifact():1525] logged artifact run-bn7ul4i2-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY0OTgwNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5MjAxMQ==', 'latestArtifact': None}} |
| 2025-01-21 21:10:44,553 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:10:44,553 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json with policy now |
| 2025-01-21 21:10:44,553 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:10:44,554 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,554 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,554 DEBUG SenderThread:2645977 [sender.py:send():379] send: telemetry |
| 2025-01-21 21:10:44,554 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,558 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,558 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,559 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,559 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,559 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,559 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,559 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,560 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,560 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,560 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,560 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,560 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,561 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,561 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,561 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,561 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,562 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,562 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,562 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,562 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:44,562 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:44,563 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 21:10:44,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:44,796 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:10:44,797 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:10:44,797 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json |
| 2025-01-21 21:10:44,797 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval |
| 2025-01-21 21:10:44,882 INFO wandb-upload_1:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/1bptd7ap-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-21 21:10:44,924 INFO wandb-upload_0:2645977 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp5qfs4pat |
| 2025-01-21 21:10:45,015 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/rrq8xh32-media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json |
| 2025-01-21 21:10:45,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:45,363 INFO SenderThread:2645977 [sender.py:send_request_log_artifact():1525] logged artifact run-bn7ul4i2-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY0OTgyNQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5MjAxNg==', 'latestArtifact': None}} |
| 2025-01-21 21:10:45,363 DEBUG SenderThread:2645977 [sender.py:send():379] send: files |
| 2025-01-21 21:10:45,363 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json with policy now |
| 2025-01-21 21:10:45,363 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:10:45,363 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:10:45,364 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:45,364 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:10:45,636 INFO wandb-upload_1:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/ahlofooo-media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json |
| 2025-01-21 21:10:45,797 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:10:46,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:46,797 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:10:47,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:48,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:49,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:50,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:51,134 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:51,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:51,792 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:10:52,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:10:52,119 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:10:52,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:53,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:54,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:55,193 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:56,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:56,281 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:10:57,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:58,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:10:59,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:00,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:01,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:02,133 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:11:02,194 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:03,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:04,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:05,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:06,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:07,119 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:11:07,119 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:11:07,192 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:11:07,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:08,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:09,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:10,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:11,195 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:12,196 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:13,137 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:11:13,196 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:13,577 DEBUG SenderThread:2645977 [sender.py:send():379] send: exit |
| 2025-01-21 21:11:13,577 INFO SenderThread:2645977 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-21 21:11:13,578 INFO SenderThread:2645977 [sender.py:send_exit():588] handling runtime: 261 |
| 2025-01-21 21:11:13,579 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:11:13,579 INFO SenderThread:2645977 [sender.py:send_exit():594] send defer |
| 2025-01-21 21:11:13,579 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:13,579 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-21 21:11:13,579 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:13,579 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-21 21:11:13,579 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-21 21:11:13,579 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:13,579 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-21 21:11:13,580 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:13,580 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-21 21:11:13,580 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-21 21:11:13,580 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:13,580 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-21 21:11:13,580 INFO HandlerThread:2645977 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-21 21:11:13,580 DEBUG SystemMonitor:2645977 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-21 21:11:13,580 INFO HandlerThread:2645977 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-21 21:11:13,580 DEBUG SystemMonitor:2645977 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-21 21:11:13,580 INFO HandlerThread:2645977 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-21 21:11:13,651 INFO HandlerThread:2645977 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-21 21:11:13,651 INFO HandlerThread:2645977 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-21 21:11:13,651 INFO HandlerThread:2645977 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-21 21:11:13,651 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:13,652 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-21 21:11:13,652 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-21 21:11:13,652 DEBUG SenderThread:2645977 [sender.py:send():379] send: stats |
| 2025-01-21 21:11:13,652 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:13,652 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-21 21:11:13,652 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:13,652 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-21 21:11:13,652 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-21 21:11:13,652 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:13,652 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-21 21:11:13,808 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/config.yaml |
| 2025-01-21 21:11:13,808 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:11:14,808 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:11:20,466 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:20,466 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:11:20,466 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-21 21:11:20,467 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:20,467 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-21 21:11:20,467 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-21 21:11:20,467 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:20,467 DEBUG SenderThread:2645977 [sender.py:send():379] send: history |
| 2025-01-21 21:11:20,467 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:20,467 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-21 21:11:20,467 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:11:20,468 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:11:20,468 DEBUG SenderThread:2645977 [sender.py:send():379] send: summary |
| 2025-01-21 21:11:20,472 INFO SenderThread:2645977 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:11:20,472 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:20,472 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-21 21:11:20,472 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-21 21:11:20,473 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:20,473 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-21 21:11:20,473 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:20,473 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-21 21:11:20,473 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-21 21:11:20,473 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:11:20,473 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:20,473 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-21 21:11:20,473 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:20,473 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-21 21:11:20,578 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:20,811 INFO Thread-12 :2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:11:23,149 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-21 21:11:23,149 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:23,149 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:23,150 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-21 21:11:23,150 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:23,150 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-21 21:11:23,150 INFO SenderThread:2645977 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-21 21:11:23,152 INFO SenderThread:2645977 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-21 21:11:23,154 INFO SenderThread:2645977 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-21 21:11:23,169 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-21 21:11:23,169 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:23,169 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-21 21:11:23,169 DEBUG SenderThread:2645977 [sender.py:send():379] send: artifact |
| 2025-01-21 21:11:23,467 INFO SenderThread:2645977 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1NzYyNTUzOQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1NzYyNTUzOQ==', 'versionIndex': 454}}} |
| 2025-01-21 21:11:23,467 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:23,467 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-21 21:11:23,467 INFO SenderThread:2645977 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-21 21:11:23,579 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:23,812 INFO SenderThread:2645977 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:11:23,812 INFO SenderThread:2645977 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files |
| 2025-01-21 21:11:23,812 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/config.yaml config.yaml |
| 2025-01-21 21:11:23,812 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json wandb-summary.json |
| 2025-01-21 21:11:23,812 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/requirements.txt requirements.txt |
| 2025-01-21 21:11:23,813 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-21 21:11:23,813 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:11:23,817 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log output.log |
| 2025-01-21 21:11:23,818 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/diff.patch diff.patch |
| 2025-01-21 21:11:23,818 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-21 21:11:23,818 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 media/videos/eval/0_eval_0_b92d3370f0db4ebdd4aa.mp4 |
| 2025-01-21 21:11:23,818 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 media/videos/eval/3_eval_3_f0cf70711762442af0f5.mp4 |
| 2025-01-21 21:11:23,818 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 media/videos/eval/4_eval_4_cd715cc078221a229a9d.mp4 |
| 2025-01-21 21:11:23,820 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 media/videos/eval/1_eval_1_a38d84554f950c6b0497.mp4 |
| 2025-01-21 21:11:23,820 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 media/videos/eval/2_eval_2_428c4186f8f49dcdf564.mp4 |
| 2025-01-21 21:11:23,820 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json media/table/eval/total_success_rate_ci_6_2033532332aee60c9249.table.json |
| 2025-01-21 21:11:23,820 INFO SenderThread:2645977 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-21 21:11:23,820 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-21 21:11:23,820 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:23,822 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:23,822 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-21 21:11:23,822 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:23,822 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-21 21:11:23,822 INFO SenderThread:2645977 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:11:24,003 INFO wandb-upload_0:2645977 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/wandb-summary.json |
| 2025-01-21 21:11:24,018 INFO wandb-upload_1:2645977 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/requirements.txt |
| 2025-01-21 21:11:24,019 INFO wandb-upload_2:2645977 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/config.yaml |
| 2025-01-21 21:11:24,071 INFO wandb-upload_4:2645977 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/files/output.log |
| 2025-01-21 21:11:24,142 INFO wandb-upload_3:2645977 [upload_job.py:push():130] Uploaded file /tmp/tmp5yjpo8npwandb/kaiul4m6-events.out.tfevents.1737511612.leonmkim-ROG-Strix-G15CS-G15CS.2645796.0 |
| 2025-01-21 21:11:24,343 INFO Thread-11 (_thread_body):2645977 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-21 21:11:24,343 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:24,343 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-21 21:11:24,343 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:24,343 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-21 21:11:24,343 INFO SenderThread:2645977 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-21 21:11:24,344 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-21 21:11:24,344 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:24,345 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-21 21:11:24,345 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:24,345 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-21 21:11:24,345 INFO SenderThread:2645977 [file_stream.py:finish():601] file stream finish called |
| 2025-01-21 21:11:24,580 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:24,633 INFO SenderThread:2645977 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-21 21:11:24,633 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-21 21:11:24,633 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:24,634 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:24,634 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-21 21:11:24,634 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:24,634 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-21 21:11:24,634 INFO SenderThread:2645977 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-21 21:11:24,634 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:11:24,634 INFO HandlerThread:2645977 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-21 21:11:24,634 DEBUG SenderThread:2645977 [sender.py:send():379] send: final |
| 2025-01-21 21:11:24,634 DEBUG SenderThread:2645977 [sender.py:send():379] send: footer |
| 2025-01-21 21:11:24,634 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:11:24,634 INFO SenderThread:2645977 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-21 21:11:24,635 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:24,635 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:24,635 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-21 21:11:24,635 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:11:24,636 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: server_info |
| 2025-01-21 21:11:24,637 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-21 21:11:24,637 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-21 21:11:24,638 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:11:24,688 DEBUG SenderThread:2645977 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:11:24,688 INFO MainThread:2645977 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-21 21:11:24,688 INFO MainThread:2645977 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-21 21:11:24,689 INFO MainThread:2645977 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-21 21:11:24,689 DEBUG HandlerThread:2645977 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-21 21:11:24,689 INFO HandlerThread:2645977 [handler.py:finish():882] shutting down handler |
| 2025-01-21 21:11:25,636 INFO WriterThread:2645977 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/bn7ul4i2/wandb/run-20250121_210651-bn7ul4i2/run-bn7ul4i2.wandb |
| 2025-01-21 21:11:25,688 INFO SenderThread:2645977 [sender.py:finish():1615] shutting down sender |
| 2025-01-21 21:11:25,688 INFO SenderThread:2645977 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:11:25,688 INFO SenderThread:2645977 [file_pusher.py:join():175] waiting for file pusher |
|
|
