| 2025-01-06 20:50:04,134 INFO StreamThr :2115858 [internal.py:wandb_internal():85] W&B internal server running at pid: 2115858, started at: 2025-01-06 20:50:04.134315 |
| 2025-01-06 20:50:04,135 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status |
| 2025-01-06 20:50:04,146 INFO WriterThread:2115858 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/run-bkletf5j.wandb |
| 2025-01-06 20:50:04,147 DEBUG SenderThread:2115858 [sender.py:send():379] send: header |
| 2025-01-06 20:50:04,155 DEBUG SenderThread:2115858 [sender.py:send():379] send: run |
| 2025-01-06 20:50:04,386 INFO SenderThread:2115858 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files |
| 2025-01-06 20:50:04,386 INFO SenderThread:2115858 [sender.py:_start_run_threads():1188] run started: bkletf5j with start time 1736214604.134792 |
| 2025-01-06 20:50:04,401 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-06 20:50:04,401 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: check_version |
| 2025-01-06 20:50:04,475 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-06 20:50:04,483 DEBUG HandlerThread:2115858 [system_info.py:__init__():26] System info init |
| 2025-01-06 20:50:04,483 DEBUG HandlerThread:2115858 [system_info.py:__init__():41] System info init done |
| 2025-01-06 20:50:04,483 INFO HandlerThread:2115858 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-06 20:50:04,483 INFO SystemMonitor:2115858 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-06 20:50:04,483 INFO HandlerThread:2115858 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-06 20:50:04,485 INFO SystemMonitor:2115858 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-06 20:50:04,485 INFO SystemMonitor:2115858 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-06 20:50:04,486 INFO SystemMonitor:2115858 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-06 20:50:04,487 INFO SystemMonitor:2115858 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-06 20:50:04,487 INFO SystemMonitor:2115858 [interfaces.py:start():188] Started network monitoring |
| 2025-01-06 20:50:04,498 DEBUG HandlerThread:2115858 [system_info.py:probe():152] Probing system |
| 2025-01-06 20:50:04,499 DEBUG HandlerThread:2115858 [system_info.py:_probe_git():137] Probing git |
| 2025-01-06 20:50:04,504 DEBUG HandlerThread:2115858 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-06 20:50:04,504 DEBUG HandlerThread:2115858 [system_info.py:probe():200] Probing system done |
| 2025-01-06 20:50:04,504 DEBUG HandlerThread:2115858 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T01:50:04.498228', 'startedAt': '2025-01-07T01:50:04.126209', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '22931875031f38fc79957f52ac8e49b1fb3d9fdc'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.57975, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4495.894, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 525.3030891418457}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-06 20:50:04,504 INFO HandlerThread:2115858 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-06 20:50:04,504 INFO HandlerThread:2115858 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-06 20:50:04,504 DEBUG HandlerThread:2115858 [system_info.py:_save_code():44] Saving code |
| 2025-01-06 20:50:04,506 DEBUG HandlerThread:2115858 [system_info.py:_save_code():67] Saving code done |
| 2025-01-06 20:50:04,506 DEBUG HandlerThread:2115858 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-06 20:50:04,573 DEBUG HandlerThread:2115858 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-06 20:50:04,573 INFO HandlerThread:2115858 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-06 20:50:04,576 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:50:04,576 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-06 20:50:04,577 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-06 20:50:04,577 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-06 20:50:04,817 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/uvzr5k1n-wandb-metadata.json |
| 2025-01-06 20:50:04,822 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-06 20:50:04,822 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-06 20:50:04,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:50:04,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:50:04,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:04,825 INFO HandlerThread:2115858 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/tb" |
| save: true |
| } |
| control { |
| relay_id: "25145b14587e4479b2cc6a16bb29b0b0" |
| } |
| _info { |
| stream_id: "bkletf5j" |
| } |
|
|
| 2025-01-06 20:50:04,826 DEBUG HandlerThread:2115858 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-06 20:50:04,874 INFO wandb-upload_1:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/eyhmgilq-code/FISH/eval_robot.py |
| 2025-01-06 20:50:04,875 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/tq8kyakw-diff.patch |
| 2025-01-06 20:50:04,978 DEBUG SenderThread:2115858 [sender.py:send():379] send: telemetry |
| 2025-01-06 20:50:04,978 DEBUG SenderThread:2115858 [sender.py:send():379] send: tbrecord |
| 2025-01-06 20:50:04,979 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:50:04,979 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 with policy live |
| 2025-01-06 20:50:05,178 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/5jzgbk5c-events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:50:05,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:05,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/diff.patch |
| 2025-01-06 20:50:05,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/code/FISH/eval_robot.py |
| 2025-01-06 20:50:05,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:50:05,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/requirements.txt |
| 2025-01-06 20:50:05,389 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-metadata.json |
| 2025-01-06 20:50:05,389 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/code |
| 2025-01-06 20:50:05,389 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/code/FISH |
| 2025-01-06 20:50:05,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:06,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:07,387 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:07,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:08,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:09,543 DEBUG SenderThread:2115858 [sender.py:send():379] send: config |
| 2025-01-06 20:50:09,543 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:09,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:10,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:11,388 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:11,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:12,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:13,389 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:13,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:14,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:14,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:15,390 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:15,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:16,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:17,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:18,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:19,391 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:19,822 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:50:19,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:50:19,865 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:19,990 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:20,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:21,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:22,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:23,392 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:23,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:24,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:25,392 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:25,643 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:25,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:26,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:27,393 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:27,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:28,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:29,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:30,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:30,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:31,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:32,833 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:33,833 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:34,822 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:50:34,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:50:34,869 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:35,833 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:35,919 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:36,395 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/config.yaml |
| 2025-01-06 20:50:36,834 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:37,834 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:38,838 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:39,838 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:40,838 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:41,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:41,839 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:42,839 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:43,840 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:44,840 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:45,840 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:46,824 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:46,840 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:47,840 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:48,841 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:49,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:50:49,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:50:49,869 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:50,841 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:51,842 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:51,916 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:52,844 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:53,631 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:50:53,631 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 with policy now |
| 2025-01-06 20:50:53,631 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:50:53,632 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:50:53,633 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:50:53,634 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:50:53,844 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:53,910 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/fnlhfe46-media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 |
| 2025-01-06 20:50:54,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:50:54,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 |
| 2025-01-06 20:50:54,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media |
| 2025-01-06 20:50:54,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos |
| 2025-01-06 20:50:54,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval |
| 2025-01-06 20:50:54,706 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/kx14jzv7-media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 |
| 2025-01-06 20:50:54,845 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:55,400 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:55,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:56,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:57,245 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:50:57,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:58,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:50:59,402 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:50:59,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:00,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:01,403 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:01,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:02,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:02,998 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:03,846 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:04,487 DEBUG SystemMonitor:2115858 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-06 20:51:04,488 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:51:04,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:51:04,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:51:04,865 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:05,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:06,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:07,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:08,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:08,923 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:09,847 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:10,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:11,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:12,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:13,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:14,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:14,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:15,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:16,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:17,408 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:51:17,650 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/8q9iiag4-events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:51:17,848 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:18,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:19,409 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:19,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:51:19,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:51:19,865 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:19,890 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:20,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:21,409 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:21,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:22,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:23,410 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:23,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:24,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:25,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:25,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:26,849 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:27,850 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:28,850 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:29,850 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:30,850 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:31,825 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:31,850 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:32,851 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:33,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:34,488 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:51:34,823 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:51:34,823 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:51:34,851 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:35,851 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:36,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:37,414 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:37,645 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:37,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:38,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:39,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:40,852 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:41,416 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:41,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:42,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:43,059 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:43,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:44,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:45,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:46,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:47,853 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:48,135 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:48,748 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:51:48,748 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 with policy now |
| 2025-01-06 20:51:48,752 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:51:48,753 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:51:48,753 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:51:48,754 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:51:48,856 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:49,017 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/0l73ku16-media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 |
| 2025-01-06 20:51:49,419 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:51:49,419 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:49,419 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 |
| 2025-01-06 20:51:49,419 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval |
| 2025-01-06 20:51:49,703 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/794wq0tz-media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 |
| 2025-01-06 20:51:49,830 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:51:49,831 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:51:49,873 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:50,857 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:51,419 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:51,858 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:52,857 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:53,420 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:53,629 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:53,858 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:54,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:55,421 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:55,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:56,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:57,421 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:57,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:58,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:51:59,313 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:51:59,314 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:51:59,314 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:51:59,314 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:51:59,422 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:51:59,422 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:51:59,422 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:51:59,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:00,859 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:01,423 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:01,860 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:02,860 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:03,424 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:03,860 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:04,489 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:52:04,489 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:04,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:52:04,826 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:52:04,873 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:05,860 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:06,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:07,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:08,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:09,637 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:09,881 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:10,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:11,426 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:11,897 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:12,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:13,427 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:13,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:14,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:14,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:15,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:16,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:17,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:18,862 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:19,429 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:19,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:52:19,826 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:52:19,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:19,921 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:20,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:21,430 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:21,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:22,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:23,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:24,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:25,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:25,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:26,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:27,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:28,863 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:29,864 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:30,601 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:52:30,601 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 with policy now |
| 2025-01-06 20:52:30,602 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:52:30,603 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:52:30,604 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:52:30,605 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:52:30,868 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:30,877 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/13jawk4c-media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 |
| 2025-01-06 20:52:31,433 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:52:31,434 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:31,434 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 |
| 2025-01-06 20:52:31,434 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval |
| 2025-01-06 20:52:31,629 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:31,737 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/poj8jw73-media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 |
| 2025-01-06 20:52:31,867 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:32,867 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:33,434 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:33,867 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:34,489 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:52:34,830 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:52:34,830 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:52:34,868 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:35,435 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:35,870 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:36,872 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:36,914 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:37,871 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:38,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:39,436 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:39,873 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:40,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:41,065 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:52:41,066 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:52:41,066 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:52:41,437 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:52:41,437 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:52:41,680 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/gmoyno9l-events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:52:41,872 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:42,098 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:42,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:43,438 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:43,880 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:44,877 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:45,873 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:46,873 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:47,128 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:47,439 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:52:47,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:48,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:49,826 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:52:49,827 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:52:49,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:50,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:51,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:52,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:52,933 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:53,874 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:54,875 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:55,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:56,875 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:57,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:58,829 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:52:58,875 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:52:59,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:00,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:01,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:02,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:03,829 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:03,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:04,489 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:53:04,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:53:04,827 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:53:04,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:05,876 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:06,877 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:07,877 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:08,877 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:08,908 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:09,877 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:10,878 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:11,878 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:12,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:13,193 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:53:13,194 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:53:13,194 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 with policy now |
| 2025-01-06 20:53:13,194 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:53:13,194 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:53:13,194 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:53:13,449 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:53:13,449 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:13,450 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 |
| 2025-01-06 20:53:13,450 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval |
| 2025-01-06 20:53:13,465 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/sbu4sowi-media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 |
| 2025-01-06 20:53:13,745 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/9g33yy8w-media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 |
| 2025-01-06 20:53:13,890 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:14,212 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:14,896 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:15,450 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:15,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:16,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:17,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:18,885 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:19,451 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:19,617 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:19,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:53:19,827 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:53:19,881 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:20,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:21,452 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:21,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:22,886 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:23,121 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:53:23,121 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:53:23,122 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:53:23,453 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:53:23,453 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:53:23,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:24,838 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:24,886 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:25,454 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:25,886 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:26,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:27,890 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:28,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:29,455 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:29,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:30,831 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:30,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:31,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:32,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:33,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:34,490 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:53:34,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:53:34,827 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:53:34,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:35,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:35,901 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:36,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:37,887 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:38,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:39,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:40,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:41,832 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:41,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:42,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:43,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:44,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:45,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:46,832 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:46,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:47,888 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:48,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:49,827 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:53:49,827 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:53:49,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:50,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:51,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:51,945 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:52,889 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:53,890 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:54,890 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:55,892 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:56,895 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:56,949 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:53:56,953 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:53:56,953 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 with policy now |
| 2025-01-06 20:53:56,956 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:53:56,956 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:53:56,957 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:53:56,958 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:53:57,225 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/e632bi05-media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 |
| 2025-01-06 20:53:57,465 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:53:57,465 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:57,465 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 |
| 2025-01-06 20:53:57,465 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval |
| 2025-01-06 20:53:57,759 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/fn7avwrs-media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 |
| 2025-01-06 20:53:57,903 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:58,897 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:53:59,465 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:53:59,900 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:00,913 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:01,466 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:01,900 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:02,733 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:54:02,908 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:03,919 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:04,490 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:54:04,841 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-06 20:54:04,841 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-06 20:54:04,916 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:05,467 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:05,917 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:06,174 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:54:06,174 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:06,174 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:06,216 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 20:54:06,216 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 20:54:06,380 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval |
| 2025-01-06 20:54:06,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table |
| 2025-01-06 20:54:06,701 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/t9sie1ly-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:54:06,714 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/qnpv54y7-events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:54:06,729 INFO wandb-upload_1:2115858 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppyoaom4f |
| 2025-01-06 20:54:06,791 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-06 20:54:06,800 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-06 20:54:06,909 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:07,141 INFO SenderThread:2115858 [sender.py:send_request_log_artifact():1525] logged artifact run-bkletf5j-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjM0MTgxNQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY1Mzc2Mg==', 'latestArtifact': None}} |
| 2025-01-06 20:54:07,141 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:54:07,141 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-06 20:54:07,141 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:54:07,142 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,143 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,143 DEBUG SenderThread:2115858 [sender.py:send():379] send: telemetry |
| 2025-01-06 20:54:07,143 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,143 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,143 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,144 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,144 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,144 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,144 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,146 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,146 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,146 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,146 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,147 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,147 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,147 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,147 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,148 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,148 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,148 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,148 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,148 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,149 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,149 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:07,149 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-06 20:54:07,376 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/0n8y3ctg-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:54:07,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:07,468 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:07,469 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:54:07,469 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json |
| 2025-01-06 20:54:07,469 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval |
| 2025-01-06 20:54:07,522 INFO wandb-upload_1:2115858 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmprhug1a3e |
| 2025-01-06 20:54:07,650 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/fkyioxav-media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json |
| 2025-01-06 20:54:07,909 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:07,963 INFO SenderThread:2115858 [sender.py:send_request_log_artifact():1525] logged artifact run-bkletf5j-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyMjM0MTg0OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMDY1Mzc2NQ==', 'latestArtifact': None}} |
| 2025-01-06 20:54:07,963 DEBUG SenderThread:2115858 [sender.py:send():379] send: files |
| 2025-01-06 20:54:07,963 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:54:07,964 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json with policy now |
| 2025-01-06 20:54:07,964 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:54:07,964 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:07,964 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:08,210 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/dxsdmipb-media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json |
| 2025-01-06 20:54:08,469 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:08,909 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:09,469 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:09,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:10,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:11,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:12,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:13,835 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:54:13,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:14,910 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:15,725 DEBUG SenderThread:2115858 [sender.py:send():379] send: exit |
| 2025-01-06 20:54:15,725 INFO SenderThread:2115858 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-06 20:54:15,726 INFO SenderThread:2115858 [sender.py:send_exit():588] handling runtime: 251 |
| 2025-01-06 20:54:15,726 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:15,726 INFO SenderThread:2115858 [sender.py:send_exit():594] send defer |
| 2025-01-06 20:54:15,726 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:15,726 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-06 20:54:15,726 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:15,726 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-06 20:54:15,726 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-06 20:54:15,726 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:15,727 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-06 20:54:15,727 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:15,727 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-06 20:54:15,727 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-06 20:54:15,727 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:15,727 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-06 20:54:15,727 INFO HandlerThread:2115858 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-06 20:54:15,727 DEBUG SystemMonitor:2115858 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-06 20:54:15,727 DEBUG SystemMonitor:2115858 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-06 20:54:15,728 INFO HandlerThread:2115858 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-06 20:54:15,728 INFO HandlerThread:2115858 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-06 20:54:15,781 INFO HandlerThread:2115858 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-06 20:54:15,781 INFO HandlerThread:2115858 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-06 20:54:15,781 INFO HandlerThread:2115858 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-06 20:54:15,781 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:15,781 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-06 20:54:15,781 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-06 20:54:15,782 DEBUG SenderThread:2115858 [sender.py:send():379] send: stats |
| 2025-01-06 20:54:15,782 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:15,782 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-06 20:54:15,782 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:15,782 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-06 20:54:15,782 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-06 20:54:15,782 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:15,782 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-06 20:54:16,472 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:17,472 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:22,179 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:54:22,179 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:54:22,179 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-06 20:54:22,180 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:22,180 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-06 20:54:22,180 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-06 20:54:22,180 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:54:22,180 DEBUG SenderThread:2115858 [sender.py:send():379] send: history |
| 2025-01-06 20:54:22,180 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-06 20:54:22,181 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:22,181 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-06 20:54:22,181 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:22,182 DEBUG SenderThread:2115858 [sender.py:send():379] send: summary |
| 2025-01-06 20:54:22,185 INFO SenderThread:2115858 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-06 20:54:22,186 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:22,186 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-06 20:54:22,186 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-06 20:54:22,186 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:22,186 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-06 20:54:22,186 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:22,186 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-06 20:54:22,190 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-06 20:54:22,301 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-06 20:54:22,302 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:22,302 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-06 20:54:22,302 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:22,302 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-06 20:54:22,475 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:22,475 INFO Thread-12 :2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/config.yaml |
| 2025-01-06 20:54:22,727 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:54:24,861 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-06 20:54:24,861 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:54:24,861 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:24,861 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-06 20:54:24,862 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:24,862 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-06 20:54:24,862 INFO SenderThread:2115858 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-06 20:54:24,864 INFO SenderThread:2115858 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-06 20:54:24,866 INFO SenderThread:2115858 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-06 20:54:24,881 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-06 20:54:24,881 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:24,881 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-06 20:54:24,881 DEBUG SenderThread:2115858 [sender.py:send():379] send: artifact |
| 2025-01-06 20:54:25,113 INFO SenderThread:2115858 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyMTkwNjU3MQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyMTk5MTI2Nw==', 'versionIndex': 427}}} |
| 2025-01-06 20:54:25,113 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:25,113 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-06 20:54:25,113 INFO SenderThread:2115858 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/config.yaml config.yaml |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json wandb-summary.json |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/requirements.txt requirements.txt |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-06 20:54:25,476 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log output.log |
| 2025-01-06 20:54:25,477 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/diff.patch diff.patch |
| 2025-01-06 20:54:25,480 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-06 20:54:25,482 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 media/videos/eval/3_eval_3_8f993ed9f4bf0eae5cf1.mp4 |
| 2025-01-06 20:54:25,483 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 media/videos/eval/4_eval_4_6a8935b2358637140fbc.mp4 |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 media/videos/eval/1_eval_1_b0cb5720b5b83600f051.mp4 |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 media/videos/eval/0_eval_0_d5629c4a8c29fc9b2a6b.mp4 |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 media/videos/eval/2_eval_2_4cf0e0208d2fc5ae5937.mp4 |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json media/table/eval/total_success_rate_ci_6_4142c490068220aa5597.table.json |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-06 20:54:25,485 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-06 20:54:25,485 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:25,485 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-06 20:54:25,486 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:25,486 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-06 20:54:25,486 INFO SenderThread:2115858 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 20:54:25,663 INFO wandb-upload_1:2115858 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/wandb-summary.json |
| 2025-01-06 20:54:25,668 INFO wandb-upload_2:2115858 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/config.yaml |
| 2025-01-06 20:54:25,705 INFO wandb-upload_3:2115858 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/requirements.txt |
| 2025-01-06 20:54:25,719 INFO wandb-upload_0:2115858 [upload_job.py:push():130] Uploaded file /tmp/tmp8_uk1vdtwandb/sacbiy0i-events.out.tfevents.1736214604.leonmkim-ROG-Strix-G15CS-G15CS.2115685.0 |
| 2025-01-06 20:54:25,728 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:54:25,728 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:54:25,769 INFO wandb-upload_4:2115858 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/files/output.log |
| 2025-01-06 20:54:25,969 INFO Thread-11 (_thread_body):2115858 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-06 20:54:25,969 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:25,969 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-06 20:54:25,969 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:25,970 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-06 20:54:25,970 INFO SenderThread:2115858 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-06 20:54:25,971 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-06 20:54:25,971 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:25,971 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-06 20:54:25,971 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:25,971 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-06 20:54:25,971 INFO SenderThread:2115858 [file_stream.py:finish():601] file stream finish called |
| 2025-01-06 20:54:26,207 INFO SenderThread:2115858 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-06 20:54:26,207 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-06 20:54:26,207 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:26,207 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-06 20:54:26,207 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:26,207 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-06 20:54:26,207 INFO SenderThread:2115858 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-06 20:54:26,207 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-06 20:54:26,207 INFO HandlerThread:2115858 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-06 20:54:26,208 DEBUG SenderThread:2115858 [sender.py:send():379] send: final |
| 2025-01-06 20:54:26,208 DEBUG SenderThread:2115858 [sender.py:send():379] send: footer |
| 2025-01-06 20:54:26,208 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: defer |
| 2025-01-06 20:54:26,208 INFO SenderThread:2115858 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-06 20:54:26,208 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:54:26,208 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:54:26,209 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-06 20:54:26,209 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: server_info |
| 2025-01-06 20:54:26,210 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-06 20:54:26,210 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-06 20:54:26,211 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-06 20:54:26,211 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-06 20:54:26,262 DEBUG SenderThread:2115858 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-06 20:54:26,262 INFO MainThread:2115858 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-06 20:54:26,263 INFO MainThread:2115858 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-06 20:54:26,263 INFO MainThread:2115858 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-06 20:54:26,263 DEBUG HandlerThread:2115858 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-06 20:54:26,263 INFO HandlerThread:2115858 [handler.py:finish():882] shutting down handler |
| 2025-01-06 20:54:27,210 INFO WriterThread:2115858 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/bkletf5j/wandb/run-20250106_205004-bkletf5j/run-bkletf5j.wandb |
| 2025-01-06 20:54:27,262 INFO SenderThread:2115858 [sender.py:finish():1615] shutting down sender |
| 2025-01-06 20:54:27,262 INFO SenderThread:2115858 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-06 20:54:27,262 INFO SenderThread:2115858 [file_pusher.py:join():175] waiting for file pusher |
|
|
