| 2025-01-07 18:23:10,073 INFO StreamThr :2211704 [internal.py:wandb_internal():85] W&B internal server running at pid: 2211704, started at: 2025-01-07 18:23:10.072732 |
| 2025-01-07 18:23:10,074 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 18:23:10,091 INFO WriterThread:2211704 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/run-no798ka2.wandb |
| 2025-01-07 18:23:10,092 DEBUG SenderThread:2211704 [sender.py:send():379] send: header |
| 2025-01-07 18:23:10,098 DEBUG SenderThread:2211704 [sender.py:send():379] send: run |
| 2025-01-07 18:23:10,371 INFO SenderThread:2211704 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files |
| 2025-01-07 18:23:10,371 INFO SenderThread:2211704 [sender.py:_start_run_threads():1188] run started: no798ka2 with start time 1736292190.073567 |
| 2025-01-07 18:23:10,390 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 18:23:10,391 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 18:23:10,469 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 18:23:10,474 DEBUG HandlerThread:2211704 [system_info.py:__init__():26] System info init |
| 2025-01-07 18:23:10,474 DEBUG HandlerThread:2211704 [system_info.py:__init__():41] System info init done |
| 2025-01-07 18:23:10,474 INFO HandlerThread:2211704 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 18:23:10,474 INFO SystemMonitor:2211704 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 18:23:10,474 INFO HandlerThread:2211704 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 18:23:10,475 INFO SystemMonitor:2211704 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 18:23:10,475 INFO SystemMonitor:2211704 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 18:23:10,476 INFO SystemMonitor:2211704 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 18:23:10,477 INFO SystemMonitor:2211704 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 18:23:10,477 INFO SystemMonitor:2211704 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 18:23:10,487 DEBUG HandlerThread:2211704 [system_info.py:probe():152] Probing system |
| 2025-01-07 18:23:10,488 DEBUG HandlerThread:2211704 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 18:23:10,493 DEBUG HandlerThread:2211704 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 18:23:10,493 DEBUG HandlerThread:2211704 [system_info.py:probe():200] Probing system done |
| 2025-01-07 18:23:10,493 DEBUG HandlerThread:2211704 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T23:23:10.487545', 'startedAt': '2025-01-07T23:23:10.063101', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.6589999999997, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4497.277, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 838.028434753418}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 18:23:10,493 INFO HandlerThread:2211704 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 18:23:10,493 INFO HandlerThread:2211704 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 18:23:10,493 DEBUG HandlerThread:2211704 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 18:23:10,495 DEBUG HandlerThread:2211704 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 18:23:10,495 DEBUG HandlerThread:2211704 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 18:23:10,556 DEBUG HandlerThread:2211704 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 18:23:10,557 INFO HandlerThread:2211704 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 18:23:10,560 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:23:10,560 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 18:23:10,560 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 18:23:10,561 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 18:23:10,807 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/tv42cjfj-wandb-metadata.json |
| 2025-01-07 18:23:10,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 18:23:10,813 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 18:23:10,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:23:10,813 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:23:10,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:10,816 INFO HandlerThread:2211704 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/tb" |
| save: true |
| } |
| control { |
| relay_id: "d6d220aa51274965a20635b41aeddf86" |
| } |
| _info { |
| stream_id: "no798ka2" |
| } |
|
|
| 2025-01-07 18:23:10,817 DEBUG HandlerThread:2211704 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 18:23:10,818 INFO wandb-upload_2:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/htub1ma4-diff.patch |
| 2025-01-07 18:23:10,842 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/n93w829l-code/FISH/eval_robot.py |
| 2025-01-07 18:23:10,916 DEBUG SenderThread:2211704 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:23:10,916 DEBUG SenderThread:2211704 [sender.py:send():379] send: tbrecord |
| 2025-01-07 18:23:10,917 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:23:10,917 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 with policy live |
| 2025-01-07 18:23:11,144 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/2j7sy6aw-events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:23:11,372 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-metadata.json |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/code/FISH/eval_robot.py |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/requirements.txt |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/diff.patch |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/code/FISH |
| 2025-01-07 18:23:11,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/code |
| 2025-01-07 18:23:11,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:12,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:13,372 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:13,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:14,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:15,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:15,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:16,495 DEBUG SenderThread:2211704 [sender.py:send():379] send: config |
| 2025-01-07 18:23:16,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:17,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:18,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:19,373 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:19,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:20,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:20,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:21,374 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:21,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:22,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:23,375 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:23,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:24,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:25,375 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:25,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:23:25,813 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:23:25,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:25,925 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:26,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:27,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:28,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:29,376 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:29,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:30,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:31,377 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:31,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:31,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:32,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:33,378 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:23:33,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:34,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:35,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:36,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:36,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:37,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:38,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:39,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:40,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:23:40,814 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:23:40,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:41,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:41,948 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:42,380 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/config.yaml |
| 2025-01-07 18:23:42,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:43,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:44,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:45,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:46,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:47,814 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:47,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:48,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:49,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:50,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:51,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:52,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:52,818 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:53,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:54,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:55,813 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:23:55,814 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:23:55,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:56,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:57,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:57,936 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:23:58,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:23:59,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:00,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:01,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:02,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:03,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:03,821 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:04,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:05,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:06,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:07,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:07,961 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:24:07,962 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:24:07,962 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 with policy now |
| 2025-01-07 18:24:07,962 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:24:07,962 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:24:07,963 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:24:08,233 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/udm3crnc-media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 |
| 2025-01-07 18:24:08,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:24:08,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 |
| 2025-01-07 18:24:08,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos |
| 2025-01-07 18:24:08,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media |
| 2025-01-07 18:24:08,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval |
| 2025-01-07 18:24:08,687 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/9p7ujrt5-media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 |
| 2025-01-07 18:24:08,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:08,964 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:09,387 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:24:09,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:10,477 DEBUG SystemMonitor:2211704 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 18:24:10,478 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:24:10,819 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:24:10,819 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:24:10,861 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:11,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:12,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:13,389 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:24:13,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:14,354 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:14,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:15,389 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:24:15,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:16,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:17,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:18,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:19,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:19,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:20,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:21,391 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:24:21,657 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/pie1o1m1-events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:24:21,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:22,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:23,392 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:24:23,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:24,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:24,950 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:25,815 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:24:25,815 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:24:25,825 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:26,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:27,393 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:24:27,826 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:28,825 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:29,825 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:30,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:30,825 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:31,825 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:32,826 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:33,826 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:34,826 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:35,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:35,827 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:36,827 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:37,827 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:38,827 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:39,827 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:40,478 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:24:40,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:24:40,816 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:24:40,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:40,943 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:41,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:42,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:43,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:44,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:45,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:46,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:46,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:47,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:48,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:49,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:50,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:51,817 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:51,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:52,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:53,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:54,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:55,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:24:55,816 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:24:55,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:56,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:56,896 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:24:57,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:58,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:24:59,829 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:00,657 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:25:00,719 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:25:00,719 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 with policy now |
| 2025-01-07 18:25:00,720 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:25:00,720 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:25:00,721 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:25:00,834 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:00,993 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/0hpy56cy-media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 |
| 2025-01-07 18:25:01,405 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:25:01,405 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:01,405 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 |
| 2025-01-07 18:25:01,406 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval |
| 2025-01-07 18:25:01,714 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/k1oiv7il-media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 |
| 2025-01-07 18:25:01,832 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:02,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:02,838 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:03,406 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:03,833 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:04,836 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:05,406 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:05,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:06,837 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:07,835 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:08,217 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:08,836 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:09,408 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:09,836 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:10,479 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:25:10,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:25:10,816 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:25:10,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:11,837 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:12,837 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:13,839 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:13,874 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:14,838 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:15,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:16,838 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:17,839 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:18,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:19,357 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:19,411 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:25:19,839 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:20,242 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:25:20,243 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:25:20,244 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:25:20,411 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:25:20,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:21,412 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:21,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:22,840 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:23,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:24,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:25,341 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:25,413 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:25,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:25:25,816 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:25:25,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:26,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:27,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:28,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:29,841 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:30,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:30,844 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:31,844 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:32,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:33,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:34,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:35,824 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:35,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:36,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:37,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:38,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:39,845 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:40,479 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:25:40,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:25:40,816 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:25:40,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:40,967 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:41,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:42,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:43,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:44,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:45,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:45,997 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:46,846 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:47,420 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:47,847 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:48,847 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:49,421 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:49,847 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:50,847 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:51,826 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:51,848 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:52,849 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:53,848 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:54,848 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:55,816 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:25:55,817 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:25:55,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:56,848 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:56,946 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:25:57,849 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:58,860 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:25:58,981 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:25:58,981 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:25:58,981 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 with policy now |
| 2025-01-07 18:25:58,982 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:25:58,982 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:25:58,999 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:25:59,256 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/o4dox3pn-media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 |
| 2025-01-07 18:25:59,424 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:25:59,425 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:25:59,425 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 |
| 2025-01-07 18:25:59,425 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval |
| 2025-01-07 18:25:59,708 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/1v94weyc-media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 |
| 2025-01-07 18:25:59,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:00,853 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:01,425 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:01,857 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:02,705 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:02,861 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:03,860 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:04,866 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:05,427 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:05,861 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:06,863 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:07,427 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:07,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:07,870 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:08,867 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:09,873 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:10,480 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:26:10,822 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:26:10,823 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:26:10,874 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:11,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:12,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:12,973 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:13,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:14,088 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:26:14,088 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:26:14,089 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:26:14,430 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:26:14,430 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:26:14,720 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/vrrxb54c-events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:26:14,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:15,430 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:15,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:16,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:17,431 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:17,874 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:17,976 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:18,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:19,431 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:19,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:20,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:21,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:22,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:23,830 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:23,875 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:24,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:25,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:26:25,820 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:26:25,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:26,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:27,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:28,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:28,913 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:29,876 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:30,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:31,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:32,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:33,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:34,689 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:34,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:35,437 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:35,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:36,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:37,438 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:37,877 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:38,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:39,831 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:39,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:40,480 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:26:40,820 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:26:40,820 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:26:40,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:41,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:42,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:43,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:44,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:44,890 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:45,878 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:46,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:47,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:48,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:49,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:50,831 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:50,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:51,879 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:52,885 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:53,443 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:53,882 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:54,016 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:26:54,016 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 with policy now |
| 2025-01-07 18:26:54,020 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:26:54,020 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:26:54,020 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:26:54,026 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:26:54,279 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/xjbh0wj3-media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 |
| 2025-01-07 18:26:54,443 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:26:54,444 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 |
| 2025-01-07 18:26:54,444 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval |
| 2025-01-07 18:26:54,745 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/nob6d7t0-media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 |
| 2025-01-07 18:26:54,884 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:55,444 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:55,833 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:26:55,833 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:26:55,890 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:55,910 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:26:56,444 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:56,885 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:57,885 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:58,892 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:26:59,445 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:26:59,891 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:00,882 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:01,446 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:01,832 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:01,895 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:02,889 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:03,892 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:04,889 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:05,895 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:06,833 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:06,896 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:07,897 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:08,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:09,155 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:27:09,155 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:27:09,156 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:27:09,449 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:27:09,449 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:27:09,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:10,481 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:27:10,835 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:27:10,835 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:27:10,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:11,449 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:11,903 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:11,951 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:12,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:13,897 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:14,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:15,451 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:15,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:16,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:17,259 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:17,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:18,898 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:19,452 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:19,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:20,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:21,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:22,832 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:22,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:23,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:24,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:25,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:27:25,823 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:27:25,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:26,899 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:27,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:27,931 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:28,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:29,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:30,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:31,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:32,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:33,833 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:33,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:34,900 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:35,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:36,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:37,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:38,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:39,833 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:39,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:40,481 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:27:40,823 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:27:40,823 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:27:40,901 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:41,903 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:42,902 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:43,902 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:44,902 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:44,959 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:45,902 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:46,903 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:47,903 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:48,906 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:48,951 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:27:48,951 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 with policy now |
| 2025-01-07 18:27:48,993 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:27:48,994 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:27:48,994 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:27:48,995 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:27:49,235 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/ytpxu9di-media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 |
| 2025-01-07 18:27:49,462 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:27:49,463 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:49,463 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 |
| 2025-01-07 18:27:49,463 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval |
| 2025-01-07 18:27:49,743 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/8uwvw4rl-media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 |
| 2025-01-07 18:27:49,911 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:49,996 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:50,915 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:51,463 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:51,906 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:52,930 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:53,464 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:53,917 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:54,914 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:55,464 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:55,499 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:27:55,828 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 18:27:55,829 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 18:27:55,923 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:56,921 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:57,465 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:27:57,927 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:58,925 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:27:59,216 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:27:59,216 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:27:59,217 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:27:59,466 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:27:59,466 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:27:59,518 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:27:59,518 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:27:59,674 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:27:59,727 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/xfeu384j-events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:27:59,921 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:28:00,007 INFO wandb-upload_0:2211704 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpmqqm3ukd |
| 2025-01-07 18:28:00,076 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 18:28:00,087 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 18:28:00,402 INFO SenderThread:2211704 [sender.py:send_request_log_artifact():1525] logged artifact run-no798ka2-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjExMzE3Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTc1NTkyOA==', 'latestArtifact': None}} |
| 2025-01-07 18:28:00,402 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:28:00,402 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now |
| 2025-01-07 18:28:00,402 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:28:00,402 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,403 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,403 DEBUG SenderThread:2211704 [sender.py:send():379] send: telemetry |
| 2025-01-07 18:28:00,403 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,407 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,407 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,408 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,408 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,408 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,408 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,409 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,409 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,409 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,409 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,410 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,410 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,410 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,410 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,410 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,411 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,411 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,411 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,411 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,412 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:00,412 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:00,412 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 18:28:00,466 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:28:00,466 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:28:00,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 18:28:00,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json |
| 2025-01-07 18:28:00,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media |
| 2025-01-07 18:28:00,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table/eval |
| 2025-01-07 18:28:00,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table |
| 2025-01-07 18:28:00,585 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/h7qe27r4-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 18:28:00,676 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/6hy7j2ag-media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json |
| 2025-01-07 18:28:00,774 INFO wandb-upload_2:2211704 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp00tumeh3 |
| 2025-01-07 18:28:00,846 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/89tlbp6u-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 18:28:00,922 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:28:01,166 INFO SenderThread:2211704 [sender.py:send_request_log_artifact():1525] logged artifact run-no798ka2-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjExMzIwMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTc1NTkzMA==', 'latestArtifact': None}} |
| 2025-01-07 18:28:01,170 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:28:01,275 DEBUG SenderThread:2211704 [sender.py:send():379] send: files |
| 2025-01-07 18:28:01,275 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json with policy now |
| 2025-01-07 18:28:01,275 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:28:01,275 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:01,275 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:01,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:28:01,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/config.yaml |
| 2025-01-07 18:28:01,467 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:28:01,531 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/175n4i24-media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json |
| 2025-01-07 18:28:01,922 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:28:02,505 DEBUG SenderThread:2211704 [sender.py:send():379] send: exit |
| 2025-01-07 18:28:02,505 INFO SenderThread:2211704 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 18:28:02,506 INFO SenderThread:2211704 [sender.py:send_exit():588] handling runtime: 292 |
| 2025-01-07 18:28:02,506 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:02,506 INFO SenderThread:2211704 [sender.py:send_exit():594] send defer |
| 2025-01-07 18:28:02,506 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:02,506 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 18:28:02,506 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:02,506 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 18:28:02,507 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 18:28:02,507 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:02,507 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 18:28:02,507 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:02,507 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 18:28:02,507 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 18:28:02,507 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:02,507 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 18:28:02,507 INFO HandlerThread:2211704 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 18:28:02,507 DEBUG SystemMonitor:2211704 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 18:28:02,507 DEBUG SystemMonitor:2211704 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 18:28:02,508 INFO HandlerThread:2211704 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 18:28:02,508 INFO HandlerThread:2211704 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 18:28:02,559 INFO HandlerThread:2211704 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 18:28:02,559 INFO HandlerThread:2211704 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 18:28:02,559 INFO HandlerThread:2211704 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 18:28:02,560 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:02,560 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 18:28:02,560 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 18:28:02,560 DEBUG SenderThread:2211704 [sender.py:send():379] send: stats |
| 2025-01-07 18:28:02,560 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:02,560 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 18:28:02,561 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:02,561 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 18:28:02,561 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 18:28:02,561 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:02,561 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 18:28:03,468 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:28:03,468 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:28:09,222 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:28:09,222 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:28:09,222 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 18:28:09,224 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:09,224 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 18:28:09,224 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 18:28:09,224 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:28:09,224 DEBUG SenderThread:2211704 [sender.py:send():379] send: history |
| 2025-01-07 18:28:09,224 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:09,224 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 18:28:09,224 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 18:28:09,225 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:09,225 DEBUG SenderThread:2211704 [sender.py:send():379] send: summary |
| 2025-01-07 18:28:09,226 INFO SenderThread:2211704 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 18:28:09,226 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:09,226 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 18:28:09,226 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 18:28:09,226 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:09,226 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 18:28:09,226 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:09,226 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 18:28:09,227 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 18:28:09,227 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 18:28:09,227 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:09,227 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 18:28:09,227 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:09,227 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 18:28:09,470 INFO Thread-12 :2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:28:09,507 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:28:12,037 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 18:28:12,037 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:28:12,037 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:12,037 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 18:28:12,038 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:12,038 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 18:28:12,038 INFO SenderThread:2211704 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 18:28:12,040 INFO SenderThread:2211704 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 18:28:12,042 INFO SenderThread:2211704 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 18:28:12,053 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 18:28:12,054 DEBUG SenderThread:2211704 [sender.py:send():379] send: artifact |
| 2025-01-07 18:28:12,056 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:12,056 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 18:28:12,238 INFO SenderThread:2211704 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTc5MjM4Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'versionIndex': 436}}} |
| 2025-01-07 18:28:12,238 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:12,238 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 18:28:12,238 INFO SenderThread:2211704 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 18:28:12,471 INFO SenderThread:2211704 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:28:12,471 INFO SenderThread:2211704 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files |
| 2025-01-07 18:28:12,471 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/config.yaml config.yaml |
| 2025-01-07 18:28:12,472 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 18:28:12,472 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/requirements.txt requirements.txt |
| 2025-01-07 18:28:12,472 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 18:28:12,472 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:28:12,476 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log output.log |
| 2025-01-07 18:28:12,476 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/diff.patch diff.patch |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 media/videos/eval/0_eval_0_c3c69d41167ae5a9086a.mp4 |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 media/videos/eval/1_eval_1_ddb2bcf1bbe47f8bb323.mp4 |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 media/videos/eval/3_eval_3_3a5d4eef35ecdfe13399.mp4 |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 media/videos/eval/2_eval_2_05d28ce3514e2b70c559.mp4 |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 media/videos/eval/4_eval_4_e8bdad0cca02a9e00142.mp4 |
| 2025-01-07 18:28:12,477 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 18:28:12,478 INFO SenderThread:2211704 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json media/table/eval/total_success_rate_ci_6_ac8537acf8bec14e3652.table.json |
| 2025-01-07 18:28:12,478 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 18:28:12,480 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:12,481 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 18:28:12,481 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:12,481 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 18:28:12,481 INFO SenderThread:2211704 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:28:12,508 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:28:12,508 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:28:12,655 INFO wandb-upload_0:2211704 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/requirements.txt |
| 2025-01-07 18:28:12,661 INFO wandb-upload_1:2211704 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/config.yaml |
| 2025-01-07 18:28:12,672 INFO wandb-upload_2:2211704 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/wandb-summary.json |
| 2025-01-07 18:28:12,688 INFO wandb-upload_4:2211704 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/files/output.log |
| 2025-01-07 18:28:12,756 INFO wandb-upload_3:2211704 [upload_job.py:push():130] Uploaded file /tmp/tmpkj6e75kawandb/9acx95ix-events.out.tfevents.1736292190.leonmkim-ROG-Strix-G15CS-G15CS.2211530.0 |
| 2025-01-07 18:28:12,957 INFO Thread-11 (_thread_body):2211704 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 18:28:12,957 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:12,957 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 18:28:12,957 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:12,957 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 18:28:12,957 INFO SenderThread:2211704 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 18:28:12,959 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 18:28:12,959 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:12,959 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 18:28:12,959 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:12,959 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 18:28:12,959 INFO SenderThread:2211704 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 18:28:13,160 INFO SenderThread:2211704 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 18:28:13,160 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 18:28:13,160 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:13,160 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 18:28:13,160 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:13,160 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 18:28:13,160 INFO SenderThread:2211704 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 18:28:13,160 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 18:28:13,160 INFO HandlerThread:2211704 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 18:28:13,161 DEBUG SenderThread:2211704 [sender.py:send():379] send: final |
| 2025-01-07 18:28:13,161 DEBUG SenderThread:2211704 [sender.py:send():379] send: footer |
| 2025-01-07 18:28:13,161 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 18:28:13,161 INFO SenderThread:2211704 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 18:28:13,161 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:28:13,161 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 18:28:13,161 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 18:28:13,162 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:28:13,162 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 18:28:13,164 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 18:28:13,164 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 18:28:13,165 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 18:28:13,210 DEBUG SenderThread:2211704 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 18:28:13,210 INFO MainThread:2211704 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 18:28:13,211 INFO MainThread:2211704 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 18:28:13,211 INFO MainThread:2211704 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 18:28:13,211 DEBUG HandlerThread:2211704 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 18:28:13,211 INFO HandlerThread:2211704 [handler.py:finish():882] shutting down handler |
| 2025-01-07 18:28:14,162 INFO WriterThread:2211704 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/no798ka2/wandb/run-20250107_182310-no798ka2/run-no798ka2.wandb |
| 2025-01-07 18:28:14,210 INFO SenderThread:2211704 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 18:28:14,210 INFO SenderThread:2211704 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 18:28:14,210 INFO SenderThread:2211704 [file_pusher.py:join():175] waiting for file pusher |
|
|
