| 2024-12-31 22:33:52,169 INFO StreamThr :1759480 [internal.py:wandb_internal():85] W&B internal server running at pid: 1759480, started at: 2024-12-31 22:33:52.168929 |
| 2024-12-31 22:33:52,170 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status |
| 2024-12-31 22:33:52,183 INFO WriterThread:1759480 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/run-3bbvaqak.wandb |
| 2024-12-31 22:33:52,183 DEBUG SenderThread:1759480 [sender.py:send():379] send: header |
| 2024-12-31 22:33:52,190 DEBUG SenderThread:1759480 [sender.py:send():379] send: run |
| 2024-12-31 22:33:52,386 INFO SenderThread:1759480 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files |
| 2024-12-31 22:33:52,386 INFO SenderThread:1759480 [sender.py:_start_run_threads():1188] run started: 3bbvaqak with start time 1735702432.170474 |
| 2024-12-31 22:33:52,402 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-31 22:33:52,402 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: check_version |
| 2024-12-31 22:33:52,479 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-31 22:33:52,484 DEBUG HandlerThread:1759480 [system_info.py:__init__():26] System info init |
| 2024-12-31 22:33:52,484 DEBUG HandlerThread:1759480 [system_info.py:__init__():41] System info init done |
| 2024-12-31 22:33:52,484 INFO HandlerThread:1759480 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-31 22:33:52,484 INFO SystemMonitor:1759480 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-31 22:33:52,484 INFO HandlerThread:1759480 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-31 22:33:52,485 INFO SystemMonitor:1759480 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-31 22:33:52,485 INFO SystemMonitor:1759480 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-31 22:33:52,486 INFO SystemMonitor:1759480 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-31 22:33:52,487 INFO SystemMonitor:1759480 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-31 22:33:52,487 INFO SystemMonitor:1759480 [interfaces.py:start():188] Started network monitoring |
| 2024-12-31 22:33:52,497 DEBUG HandlerThread:1759480 [system_info.py:probe():152] Probing system |
| 2024-12-31 22:33:52,498 DEBUG HandlerThread:1759480 [system_info.py:_probe_git():137] Probing git |
| 2024-12-31 22:33:52,503 DEBUG HandlerThread:1759480 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-31 22:33:52,503 DEBUG HandlerThread:1759480 [system_info.py:probe():200] Probing system done |
| 2024-12-31 22:33:52,503 DEBUG HandlerThread:1759480 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T03:33:52.497155', 'startedAt': '2025-01-01T03:33:52.161539', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3560.59375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.004, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4496.64, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 560.3900833129883}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-31 22:33:52,503 INFO HandlerThread:1759480 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-31 22:33:52,503 INFO HandlerThread:1759480 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-31 22:33:52,503 DEBUG HandlerThread:1759480 [system_info.py:_save_code():44] Saving code |
| 2024-12-31 22:33:52,505 DEBUG HandlerThread:1759480 [system_info.py:_save_code():67] Saving code done |
| 2024-12-31 22:33:52,505 DEBUG HandlerThread:1759480 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-31 22:33:52,567 DEBUG HandlerThread:1759480 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-31 22:33:52,567 INFO HandlerThread:1759480 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-31 22:33:52,570 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:33:52,570 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-31 22:33:52,570 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-31 22:33:52,570 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-31 22:33:52,800 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-31 22:33:52,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-31 22:33:52,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:52,802 DEBUG SenderThread:1759480 [sender.py:send():379] send: telemetry |
| 2024-12-31 22:33:52,820 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/g166iolm-wandb-metadata.json |
| 2024-12-31 22:33:52,831 INFO wandb-upload_2:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/iz96odku-diff.patch |
| 2024-12-31 22:33:52,849 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:33:52,849 INFO HandlerThread:1759480 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/tb" |
| save: true |
| } |
| control { |
| relay_id: "bf3d693ffd2549f08dea2c0f264a209a" |
| } |
| _info { |
| stream_id: "3bbvaqak" |
| } |
|
|
| 2024-12-31 22:33:52,849 DEBUG HandlerThread:1759480 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-31 22:33:52,850 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:33:52,854 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/z57m9fzd-code/FISH/eval_robot.py |
| 2024-12-31 22:33:53,004 DEBUG SenderThread:1759480 [sender.py:send():379] send: tbrecord |
| 2024-12-31 22:33:53,004 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:33:53,004 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 with policy live |
| 2024-12-31 22:33:53,188 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/dm0fawzm-events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/diff.patch |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/code/FISH/eval_robot.py |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-metadata.json |
| 2024-12-31 22:33:53,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/requirements.txt |
| 2024-12-31 22:33:53,389 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/code/FISH |
| 2024-12-31 22:33:53,389 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/code |
| 2024-12-31 22:33:53,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:54,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:55,387 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:33:55,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:56,746 DEBUG SenderThread:1759480 [sender.py:send():379] send: config |
| 2024-12-31 22:33:56,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:57,757 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:33:57,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:58,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:33:59,388 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:33:59,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:00,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:01,389 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:01,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:02,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:02,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:03,389 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:03,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:04,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:05,390 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:05,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:06,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:07,391 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:07,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:34:07,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:34:07,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:07,910 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:08,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:09,391 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:09,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:10,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:11,392 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:11,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:12,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:13,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:13,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:14,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:15,394 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:15,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:16,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:17,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:18,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:18,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:19,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:20,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:21,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:22,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:34:22,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:34:22,845 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:23,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:23,922 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:24,396 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/config.yaml |
| 2024-12-31 22:34:24,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:25,397 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:25,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:26,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:27,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:28,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:29,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:29,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:30,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:31,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:32,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:33,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:34,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:34,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:35,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:36,807 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:37,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:34:37,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:34:37,845 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:38,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:39,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:39,872 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:40,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:41,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:42,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:43,401 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:43,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:44,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:45,541 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:45,769 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:34:45,769 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 with policy now |
| 2024-12-31 22:34:45,769 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:34:45,771 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:34:45,771 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:34:45,771 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:34:45,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:46,013 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/holiaaxl-media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 |
| 2024-12-31 22:34:46,402 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:34:46,403 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 |
| 2024-12-31 22:34:46,403 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media |
| 2024-12-31 22:34:46,403 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos |
| 2024-12-31 22:34:46,403 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval |
| 2024-12-31 22:34:46,683 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/9qjwdrvb-media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 |
| 2024-12-31 22:34:46,808 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:47,403 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:47,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:48,809 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:49,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:50,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:51,365 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:51,404 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:51,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:52,487 DEBUG SystemMonitor:1759480 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-31 22:34:52,488 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:34:52,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:34:52,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:34:52,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:53,405 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:53,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:54,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:55,405 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:34:55,699 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/d371h7p7-events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:34:55,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:56,653 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:34:56,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:57,406 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:57,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:58,811 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:34:59,407 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:34:59,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:00,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:01,408 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:01,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:02,262 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:02,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:03,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:04,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:05,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:06,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:07,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:35:07,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:35:07,845 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:07,862 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:08,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:09,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:10,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:11,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:12,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:13,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:13,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:14,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:15,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:16,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:17,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:18,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:18,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:19,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:20,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:21,814 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:22,488 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:35:22,801 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:35:22,801 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:35:22,849 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:23,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:23,880 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:24,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:25,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:26,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:27,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:28,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:29,363 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:29,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:30,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:31,417 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:31,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:32,817 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:33,817 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:34,572 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:34,820 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:35:34,820 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 with policy now |
| 2024-12-31 22:35:34,823 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:35:34,824 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:35:34,824 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:35:34,827 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:35:34,828 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:35,064 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/kofetkye-media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 |
| 2024-12-31 22:35:35,418 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:35,418 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:35:35,418 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 |
| 2024-12-31 22:35:35,419 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval |
| 2024-12-31 22:35:35,709 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/j629ah60-media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 |
| 2024-12-31 22:35:35,829 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:36,830 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:37,419 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:37,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:35:37,812 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:35:37,853 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:38,832 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:39,420 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:39,827 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:40,413 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:40,824 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:41,832 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:42,829 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:43,421 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:43,828 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:44,828 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:45,812 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:45,828 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:46,831 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:47,839 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:48,839 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:49,834 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:50,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:50,830 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:51,834 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:52,489 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:35:52,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:35:52,804 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:35:52,849 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:53,833 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:54,837 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:55,834 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:55,897 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:35:56,836 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:57,833 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:58,265 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:35:58,265 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:35:58,265 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:35:58,426 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:35:58,426 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:35:58,679 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/h3wua1zg-events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:35:58,838 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:35:59,426 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:35:59,842 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:00,842 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:01,261 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:01,841 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:02,840 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:03,428 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:03,841 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:05,035 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:05,428 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:06,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:06,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:07,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:07,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:36:07,802 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:36:08,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:09,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:10,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:11,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:11,890 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:12,027 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:13,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:13,432 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:14,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:15,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:16,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:17,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:17,805 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:18,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:19,028 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:19,434 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:20,029 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:21,029 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:21,434 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:22,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:22,489 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:36:22,802 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:36:22,802 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:36:22,888 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:23,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:24,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:25,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:26,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:27,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:28,030 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:28,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:29,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:30,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:31,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:32,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:33,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:33,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:34,032 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:35,031 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:36,032 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:37,032 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:37,817 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:36:37,817 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:36:38,035 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:38,308 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:36:38,309 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 with policy now |
| 2024-12-31 22:36:38,313 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:36:38,314 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:36:38,314 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:36:38,314 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:36:38,440 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:36:38,440 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 |
| 2024-12-31 22:36:38,440 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval |
| 2024-12-31 22:36:38,547 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/d9zzwm60-media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 |
| 2024-12-31 22:36:38,827 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/s3pgh7ip-media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 |
| 2024-12-31 22:36:39,045 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:39,343 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:39,440 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:40,047 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:41,045 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:42,046 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:43,046 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:43,441 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:44,047 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:44,737 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:45,046 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:46,048 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:47,056 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:47,443 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:48,057 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:49,050 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:49,111 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:36:49,111 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:36:49,111 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:36:49,444 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:36:49,444 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:36:49,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:50,058 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:51,057 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:51,444 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:52,070 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:52,490 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:36:52,806 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:36:52,807 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:36:53,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:53,445 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:54,067 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:55,059 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:55,321 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:36:55,446 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:56,062 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:57,060 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:57,446 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:36:58,060 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:36:59,061 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:00,061 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:00,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:01,061 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:02,061 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:03,062 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:04,062 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:05,062 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:05,813 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:06,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:07,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:07,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:37:07,803 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:37:08,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:09,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:10,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:10,920 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:11,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:12,063 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:13,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:14,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:15,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:15,453 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:16,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:16,494 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:17,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:18,066 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:19,064 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:20,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:21,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:21,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:22,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:22,490 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:37:22,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:37:22,803 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:37:23,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:24,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:25,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:26,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:26,914 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:27,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:28,065 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:29,066 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:30,081 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:31,076 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:31,148 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:37:31,149 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 with policy now |
| 2024-12-31 22:37:31,149 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:37:31,150 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:37:31,150 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:37:31,158 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:37:31,389 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/ns4oe1jc-media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 |
| 2024-12-31 22:37:31,458 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:31,458 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:37:31,459 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 |
| 2024-12-31 22:37:31,459 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval |
| 2024-12-31 22:37:31,745 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/mn74s48n-media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 |
| 2024-12-31 22:37:32,071 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:32,185 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:33,076 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:33,459 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:34,080 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:35,078 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:35,460 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:36,075 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:37,076 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:37,497 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:37,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:37:37,803 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:37:38,077 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:39,078 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:39,461 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:40,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:41,094 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:42,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:42,617 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:43,084 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:43,174 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:37:43,175 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:37:43,175 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:37:43,462 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:43,462 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:37:43,463 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:37:43,705 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/mdfu3xib-events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:37:44,086 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:45,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:45,463 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:46,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:47,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:47,464 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:47,816 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:48,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:49,085 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:50,086 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:51,085 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:51,465 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:37:52,085 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:52,491 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:37:52,803 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:37:52,804 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:37:52,934 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:53,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:54,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:55,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:56,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:57,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:58,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:37:58,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:37:59,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:00,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:01,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:01,468 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:02,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:03,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:03,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:04,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:05,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:05,470 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:06,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:07,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:07,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:38:07,804 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:38:08,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:08,907 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:09,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:10,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:11,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:12,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:13,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:14,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:14,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:15,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:16,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:17,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:18,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:19,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:19,815 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:20,092 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:21,091 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:22,094 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:22,491 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:38:22,804 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 22:38:22,804 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 22:38:23,092 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:24,092 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:25,095 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:25,453 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:25,477 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:25,734 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:38:25,734 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_72049e080103157b9708.mp4 with policy now |
| 2024-12-31 22:38:25,735 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:38:25,735 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:38:25,735 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:25,737 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:25,978 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/neu09czi-media/videos/eval/4_eval_4_72049e080103157b9708.mp4 |
| 2024-12-31 22:38:26,095 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:26,477 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:26,477 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/4_eval_4_72049e080103157b9708.mp4 |
| 2024-12-31 22:38:26,477 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval |
| 2024-12-31 22:38:26,799 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/5yizbm4v-media/videos/eval/4_eval_4_72049e080103157b9708.mp4 |
| 2024-12-31 22:38:27,123 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:27,477 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:28,102 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:29,116 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:30,106 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:31,118 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:31,445 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:31,479 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:32,103 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:33,114 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:33,479 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:34,113 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:35,120 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:35,240 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:38:35,240 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:35,241 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:35,352 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 22:38:35,352 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 22:38:35,480 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:35,480 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:38:35,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2024-12-31 22:38:35,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media |
| 2024-12-31 22:38:35,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval |
| 2024-12-31 22:38:35,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table |
| 2024-12-31 22:38:35,564 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:38:35,664 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/c8l6zeit-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2024-12-31 22:38:35,804 INFO wandb-upload_1:1759480 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpvhsttq9_ |
| 2024-12-31 22:38:35,976 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 22:38:35,987 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 22:38:36,104 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:36,235 INFO SenderThread:1759480 [sender.py:send_request_log_artifact():1525] logged artifact run-3bbvaqak-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDE2NzU1NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3MzkwNQ==', 'latestArtifact': None}} |
| 2024-12-31 22:38:36,235 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:38:36,235 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2024-12-31 22:38:36,235 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:38:36,235 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,236 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,236 DEBUG SenderThread:1759480 [sender.py:send():379] send: telemetry |
| 2024-12-31 22:38:36,236 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,239 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,240 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,240 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,240 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,240 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,241 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,241 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,241 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,241 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,242 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,242 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,242 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,242 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,243 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,243 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,243 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,245 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,246 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,247 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,247 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:36,247 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:36,247 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 22:38:36,480 INFO wandb-upload_2:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/dxc497fk-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2024-12-31 22:38:36,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:36,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:38:36,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json |
| 2024-12-31 22:38:36,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval |
| 2024-12-31 22:38:36,603 INFO wandb-upload_1:1759480 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpqm7wnaml |
| 2024-12-31 22:38:36,678 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/o25kszuh-media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json |
| 2024-12-31 22:38:37,047 INFO SenderThread:1759480 [sender.py:send_request_log_artifact():1525] logged artifact run-3bbvaqak-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDE2NzU2OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3MzkwOA==', 'latestArtifact': None}} |
| 2024-12-31 22:38:37,047 DEBUG SenderThread:1759480 [sender.py:send():379] send: files |
| 2024-12-31 22:38:37,047 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json with policy now |
| 2024-12-31 22:38:37,047 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:37,047 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:38:37,047 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:37,048 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:37,105 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:37,205 DEBUG SenderThread:1759480 [sender.py:send():379] send: exit |
| 2024-12-31 22:38:37,205 INFO SenderThread:1759480 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-31 22:38:37,206 INFO SenderThread:1759480 [sender.py:send_exit():588] handling runtime: 284 |
| 2024-12-31 22:38:37,206 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:37,206 INFO SenderThread:1759480 [sender.py:send_exit():594] send defer |
| 2024-12-31 22:38:37,206 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:37,206 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-31 22:38:37,206 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:37,206 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-31 22:38:37,206 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-31 22:38:37,206 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:37,207 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-31 22:38:37,207 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:37,207 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-31 22:38:37,207 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-31 22:38:37,207 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:37,207 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-31 22:38:37,207 INFO HandlerThread:1759480 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-31 22:38:37,207 INFO HandlerThread:1759480 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-31 22:38:37,207 INFO HandlerThread:1759480 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-31 22:38:37,208 DEBUG SystemMonitor:1759480 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-31 22:38:37,208 DEBUG SystemMonitor:1759480 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-31 22:38:37,260 INFO HandlerThread:1759480 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-31 22:38:37,260 INFO HandlerThread:1759480 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-31 22:38:37,260 INFO HandlerThread:1759480 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-31 22:38:37,261 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:37,261 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-31 22:38:37,261 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-31 22:38:37,261 DEBUG SenderThread:1759480 [sender.py:send():379] send: stats |
| 2024-12-31 22:38:37,261 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:37,261 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-31 22:38:37,261 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:37,261 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-31 22:38:37,261 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-31 22:38:37,261 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:37,261 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-31 22:38:37,325 INFO wandb-upload_2:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/33tyrnx4-media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json |
| 2024-12-31 22:38:37,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:37,481 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:39,482 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:42,483 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/config.yaml |
| 2024-12-31 22:38:44,248 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 22:38:44,248 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:44,248 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-31 22:38:44,249 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:44,249 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-31 22:38:44,249 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-31 22:38:44,249 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 22:38:44,249 DEBUG SenderThread:1759480 [sender.py:send():379] send: history |
| 2024-12-31 22:38:44,249 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 22:38:44,250 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:44,250 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-31 22:38:44,250 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:44,251 DEBUG SenderThread:1759480 [sender.py:send():379] send: summary |
| 2024-12-31 22:38:44,251 INFO SenderThread:1759480 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 22:38:44,251 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:44,251 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-31 22:38:44,251 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-31 22:38:44,251 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:44,251 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-31 22:38:44,251 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:44,251 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-31 22:38:44,252 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-31 22:38:44,252 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 22:38:44,252 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:44,252 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-31 22:38:44,252 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:44,252 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-31 22:38:44,484 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:45,207 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 22:38:45,309 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-31 22:38:45,309 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 22:38:45,309 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:45,309 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-31 22:38:45,310 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:45,310 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-31 22:38:45,310 INFO SenderThread:1759480 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-31 22:38:45,312 INFO SenderThread:1759480 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-31 22:38:45,314 INFO SenderThread:1759480 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-31 22:38:45,328 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-31 22:38:45,329 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:45,329 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-31 22:38:45,329 DEBUG SenderThread:1759480 [sender.py:send():379] send: artifact |
| 2024-12-31 22:38:45,485 INFO Thread-12 :1759480 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:45,775 INFO wandb-upload_1:1759480 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1lk1rrzv |
| 2024-12-31 22:38:45,775 INFO wandb-upload_2:1759480 [upload_job.py:push():85] Skipped uploading /tmp/tmpib2suzf2/wandb-job.json |
| 2024-12-31 22:38:45,897 INFO wandb-upload_0:1759480 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpsrixm6e7 |
| 2024-12-31 22:38:46,207 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 22:38:46,479 INFO SenderThread:1759480 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDE2NzY3NQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDE2MzgxOA==', 'versionIndex': 397}}} |
| 2024-12-31 22:38:46,480 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:46,480 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-31 22:38:46,480 INFO SenderThread:1759480 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-31 22:38:46,485 INFO SenderThread:1759480 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files |
| 2024-12-31 22:38:46,485 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/config.yaml config.yaml |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json wandb-summary.json |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/requirements.txt requirements.txt |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log output.log |
| 2024-12-31 22:38:46,486 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/diff.patch diff.patch |
| 2024-12-31 22:38:46,489 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-31 22:38:46,491 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 media/videos/eval/2_eval_2_69a94bb80ce32994374e.mp4 |
| 2024-12-31 22:38:46,491 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 media/videos/eval/0_eval_0_c2ff683945f18b252db9.mp4 |
| 2024-12-31 22:38:46,491 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/4_eval_4_72049e080103157b9708.mp4 media/videos/eval/4_eval_4_72049e080103157b9708.mp4 |
| 2024-12-31 22:38:46,491 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 media/videos/eval/3_eval_3_e19838f640f8887956ae.mp4 |
| 2024-12-31 22:38:46,491 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 media/videos/eval/1_eval_1_b0a9c00b6ce785ea7915.mp4 |
| 2024-12-31 22:38:46,493 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json media/table/eval/total_success_rate_ci_6_6372d751b56b6efde046.table.json |
| 2024-12-31 22:38:46,493 INFO SenderThread:1759480 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2024-12-31 22:38:46,493 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-31 22:38:46,493 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 22:38:46,495 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:46,495 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-31 22:38:46,495 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:46,495 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-31 22:38:46,495 INFO SenderThread:1759480 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 22:38:46,669 INFO wandb-upload_2:1759480 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/wandb-summary.json |
| 2024-12-31 22:38:46,697 INFO wandb-upload_0:1759480 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/requirements.txt |
| 2024-12-31 22:38:46,741 INFO wandb-upload_1:1759480 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/config.yaml |
| 2024-12-31 22:38:46,744 INFO wandb-upload_4:1759480 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/files/output.log |
| 2024-12-31 22:38:46,822 INFO wandb-upload_3:1759480 [upload_job.py:push():130] Uploaded file /tmp/tmpn9kjcegrwandb/nnco23cm-events.out.tfevents.1735702432.leonmkim-ROG-Strix-G15CS-G15CS.1759289.0 |
| 2024-12-31 22:38:47,023 INFO Thread-11 (_thread_body):1759480 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-31 22:38:47,023 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:47,023 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-31 22:38:47,023 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:47,023 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-31 22:38:47,023 INFO SenderThread:1759480 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-31 22:38:47,025 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-31 22:38:47,025 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:47,025 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-31 22:38:47,025 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:47,025 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-31 22:38:47,025 INFO SenderThread:1759480 [file_stream.py:finish():601] file stream finish called |
| 2024-12-31 22:38:47,085 INFO SenderThread:1759480 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-31 22:38:47,086 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-31 22:38:47,086 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:47,086 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-31 22:38:47,086 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:47,086 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-31 22:38:47,086 INFO SenderThread:1759480 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-31 22:38:47,086 DEBUG SenderThread:1759480 [sender.py:send():379] send: final |
| 2024-12-31 22:38:47,086 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 22:38:47,086 DEBUG SenderThread:1759480 [sender.py:send():379] send: footer |
| 2024-12-31 22:38:47,086 INFO HandlerThread:1759480 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-31 22:38:47,087 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 22:38:47,087 INFO SenderThread:1759480 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-31 22:38:47,087 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 22:38:47,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-31 22:38:47,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 22:38:47,088 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 22:38:47,088 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-31 22:38:47,088 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: server_info |
| 2024-12-31 22:38:47,089 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-31 22:38:47,090 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 22:38:47,141 DEBUG SenderThread:1759480 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 22:38:47,141 INFO MainThread:1759480 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-31 22:38:47,141 INFO MainThread:1759480 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-31 22:38:47,142 INFO MainThread:1759480 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-31 22:38:47,142 DEBUG HandlerThread:1759480 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-31 22:38:47,142 INFO HandlerThread:1759480 [handler.py:finish():882] shutting down handler |
| 2024-12-31 22:38:48,088 INFO WriterThread:1759480 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/3bbvaqak/wandb/run-20241231_223352-3bbvaqak/run-3bbvaqak.wandb |
| 2024-12-31 22:38:48,141 INFO SenderThread:1759480 [sender.py:finish():1615] shutting down sender |
| 2024-12-31 22:38:48,141 INFO SenderThread:1759480 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 22:38:48,141 INFO SenderThread:1759480 [file_pusher.py:join():175] waiting for file pusher |
|
|
