| 2025-01-21 21:30:45,832 INFO StreamThr :2651916 [internal.py:wandb_internal():85] W&B internal server running at pid: 2651916, started at: 2025-01-21 21:30:45.832388 |
| 2025-01-21 21:30:45,834 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status |
| 2025-01-21 21:30:45,844 INFO WriterThread:2651916 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/run-pkbouru1.wandb |
| 2025-01-21 21:30:45,844 DEBUG SenderThread:2651916 [sender.py:send():379] send: header |
| 2025-01-21 21:30:45,852 DEBUG SenderThread:2651916 [sender.py:send():379] send: run |
| 2025-01-21 21:30:46,167 INFO SenderThread:2651916 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files |
| 2025-01-21 21:30:46,167 INFO SenderThread:2651916 [sender.py:_start_run_threads():1188] run started: pkbouru1 with start time 1737513045.832786 |
| 2025-01-21 21:30:46,186 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-21 21:30:46,186 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: check_version |
| 2025-01-21 21:30:46,262 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-21 21:30:46,267 DEBUG HandlerThread:2651916 [system_info.py:__init__():26] System info init |
| 2025-01-21 21:30:46,267 DEBUG HandlerThread:2651916 [system_info.py:__init__():41] System info init done |
| 2025-01-21 21:30:46,267 INFO HandlerThread:2651916 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-21 21:30:46,267 INFO SystemMonitor:2651916 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-21 21:30:46,268 INFO SystemMonitor:2651916 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-21 21:30:46,268 INFO HandlerThread:2651916 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-21 21:30:46,269 INFO SystemMonitor:2651916 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-21 21:30:46,269 INFO SystemMonitor:2651916 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-21 21:30:46,270 INFO SystemMonitor:2651916 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-21 21:30:46,270 INFO SystemMonitor:2651916 [interfaces.py:start():188] Started network monitoring |
| 2025-01-21 21:30:46,281 DEBUG HandlerThread:2651916 [system_info.py:probe():152] Probing system |
| 2025-01-21 21:30:46,283 DEBUG HandlerThread:2651916 [system_info.py:_probe_git():137] Probing git |
| 2025-01-21 21:30:46,288 DEBUG HandlerThread:2651916 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-21 21:30:46,288 DEBUG HandlerThread:2651916 [system_info.py:probe():200] Probing system done |
| 2025-01-21 21:30:46,288 DEBUG HandlerThread:2651916 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-22T02:30:46.281548', 'startedAt': '2025-01-22T02:30:45.820874', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'dc9da9fa0ef2996c7fec8d105e7933e728cd952f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.7545, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4500.003, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.003, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 381.4694061279297}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-21 21:30:46,288 INFO HandlerThread:2651916 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-21 21:30:46,288 INFO HandlerThread:2651916 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-21 21:30:46,288 DEBUG HandlerThread:2651916 [system_info.py:_save_code():44] Saving code |
| 2025-01-21 21:30:46,289 DEBUG HandlerThread:2651916 [system_info.py:_save_code():67] Saving code done |
| 2025-01-21 21:30:46,289 DEBUG HandlerThread:2651916 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-21 21:30:46,352 DEBUG HandlerThread:2651916 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-21 21:30:46,353 INFO HandlerThread:2651916 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-21 21:30:46,360 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:30:46,360 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-21 21:30:46,361 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-21 21:30:46,361 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-21 21:30:46,589 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-21 21:30:46,589 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-21 21:30:46,590 DEBUG SenderThread:2651916 [sender.py:send():379] send: telemetry |
| 2025-01-21 21:30:46,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:46,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:30:46,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:30:46,633 INFO HandlerThread:2651916 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/tb" |
| save: true |
| } |
| control { |
| relay_id: "f6d28eb1c12a46838862bea7e051a09a" |
| } |
| _info { |
| stream_id: "pkbouru1" |
| } |
|
|
| 2025-01-21 21:30:46,633 DEBUG HandlerThread:2651916 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-21 21:30:46,652 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/ea9nclmn-diff.patch |
| 2025-01-21 21:30:46,658 INFO wandb-upload_1:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/f2w68x3r-code/FISH/eval_robot.py |
| 2025-01-21 21:30:46,659 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/tpdhf4pz-wandb-metadata.json |
| 2025-01-21 21:30:46,718 DEBUG SenderThread:2651916 [sender.py:send():379] send: tbrecord |
| 2025-01-21 21:30:46,718 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:30:46,718 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 with policy live |
| 2025-01-21 21:30:46,897 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/dfpwy5ze-events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:30:47,169 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-metadata.json |
| 2025-01-21 21:30:47,169 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/requirements.txt |
| 2025-01-21 21:30:47,169 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:30:47,169 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/code/FISH/eval_robot.py |
| 2025-01-21 21:30:47,170 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/diff.patch |
| 2025-01-21 21:30:47,170 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:30:47,170 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/code/FISH |
| 2025-01-21 21:30:47,170 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/code |
| 2025-01-21 21:30:47,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:48,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:49,168 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:30:49,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:50,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:51,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:30:51,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:52,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:53,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:54,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:55,016 DEBUG SenderThread:2651916 [sender.py:send():379] send: config |
| 2025-01-21 21:30:55,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:56,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:30:56,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:57,172 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:30:57,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:58,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:30:59,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:00,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:01,173 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:01,589 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:31:01,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:31:01,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:01,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:02,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:03,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:04,593 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:05,174 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:05,593 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:06,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:07,175 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:07,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:07,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:08,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:09,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:10,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:11,176 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:11,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:12,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:12,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:13,594 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:14,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:15,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:16,589 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:31:16,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:31:16,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:17,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:17,719 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:18,177 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/config.yaml |
| 2025-01-21 21:31:18,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:19,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:20,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:21,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:22,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:23,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:23,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:24,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:25,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:26,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:27,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:28,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:28,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:29,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:30,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:31,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:31:31,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:31:31,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:32,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:33,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:33,677 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:34,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:35,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:36,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:37,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:38,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:39,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:39,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:40,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:41,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:42,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:43,184 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:43,609 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:44,573 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:31:44,574 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:31:44,574 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 with policy now |
| 2025-01-21 21:31:44,575 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:31:44,575 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:31:44,575 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:31:44,598 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:44,833 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/83plspd5-media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos |
| 2025-01-21 21:31:45,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval |
| 2025-01-21 21:31:45,469 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/p2eq6pp7-media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 |
| 2025-01-21 21:31:45,584 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:45,598 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:46,270 DEBUG SystemMonitor:2651916 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-21 21:31:46,271 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:31:46,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:31:46,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:31:46,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:47,185 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:47,599 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:48,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:49,186 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:49,600 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:50,600 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:50,953 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:51,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:52,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:53,187 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:53,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:54,188 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:31:54,489 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/oadx6w7s-events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:31:54,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:55,188 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:55,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:56,025 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:31:56,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:57,189 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:57,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:58,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:31:59,190 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:31:59,601 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:00,602 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:01,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:32:01,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:32:01,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:01,705 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:02,602 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:03,602 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:04,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:05,602 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:06,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:07,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:07,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:08,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:09,604 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:10,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:11,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:12,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:12,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:13,604 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:14,605 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:15,605 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:16,271 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:32:16,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:32:16,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:32:16,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:17,195 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:17,605 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:17,662 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:18,605 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:19,196 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:19,605 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:20,606 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:21,606 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:22,606 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:23,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:23,606 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:24,608 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:25,607 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:26,607 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:27,607 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:28,592 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:28,607 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:29,608 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:30,608 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:31,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:32:31,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:32:31,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:32,608 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:33,611 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:33,714 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:33,757 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:32:33,757 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:32:33,758 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 with policy now |
| 2025-01-21 21:32:33,758 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:32:33,758 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:32:33,759 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:32:33,981 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/ig7c3sos-media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 |
| 2025-01-21 21:32:34,201 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:32:34,201 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 |
| 2025-01-21 21:32:34,201 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval |
| 2025-01-21 21:32:34,484 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/mj0ozn2x-media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 |
| 2025-01-21 21:32:34,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:35,201 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:35,615 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:36,617 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:37,618 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:38,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:39,203 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:39,417 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:39,621 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:40,615 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:41,203 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:41,615 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:42,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:43,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:44,593 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:44,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:45,618 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:46,272 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:32:46,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:32:46,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:32:46,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:47,617 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:48,617 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:49,063 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:32:49,063 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:32:49,063 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:32:49,206 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:32:49,206 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:32:49,618 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:50,537 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:50,617 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:51,206 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:51,617 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:52,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:53,207 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:53,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:54,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:55,208 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:32:55,596 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:32:55,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:56,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:57,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:58,619 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:32:59,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:00,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:01,590 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:33:01,590 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:33:01,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:01,710 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:02,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:03,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:04,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:05,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:06,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:07,593 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:07,620 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:08,628 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:08,984 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:33:08,984 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 with policy now |
| 2025-01-21 21:33:08,991 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:33:08,991 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:33:08,992 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:33:08,992 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:33:09,198 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/zvcmr5ax-media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 |
| 2025-01-21 21:33:09,212 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:09,213 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:33:09,213 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 |
| 2025-01-21 21:33:09,213 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval |
| 2025-01-21 21:33:09,477 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/ty0mj2be-media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 |
| 2025-01-21 21:33:09,631 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:10,625 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:11,213 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:11,627 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:12,598 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:12,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:13,627 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:14,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:15,214 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:15,625 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:16,272 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:33:16,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:33:16,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:33:16,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:17,215 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:17,627 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:17,720 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:18,628 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:19,628 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:20,628 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:21,627 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:22,632 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:23,595 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:23,636 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:24,636 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:25,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:26,637 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:27,219 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:33:27,469 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/ol83ayog-events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:33:27,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:27,887 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:33:27,887 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:33:27,887 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:33:28,219 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:33:28,646 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:28,993 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:29,219 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:29,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:30,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:31,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:33:31,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:33:31,639 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:32,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:33,221 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:33,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:34,639 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:34,732 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:35,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:36,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:37,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:38,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:39,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:40,598 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:40,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:41,640 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:42,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:43,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:44,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:45,598 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:45,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:46,273 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:33:46,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:33:46,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:33:46,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:47,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:48,641 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:49,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:50,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:50,742 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:51,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:52,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:53,227 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:33:53,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:54,642 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:55,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:56,437 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:33:56,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:57,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:58,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:33:59,643 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:00,644 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:01,591 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:34:01,591 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:34:01,644 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:01,744 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:02,644 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:03,644 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:04,644 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:05,653 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:06,650 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:06,761 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:34:06,761 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 with policy now |
| 2025-01-21 21:34:06,761 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:34:06,763 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:06,764 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:34:06,765 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:34:06,770 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:34:06,990 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/x2vu43wd-media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 |
| 2025-01-21 21:34:07,232 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:07,232 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:34:07,232 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 |
| 2025-01-21 21:34:07,232 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval |
| 2025-01-21 21:34:07,504 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/im2a57kj-media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 |
| 2025-01-21 21:34:07,650 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:08,649 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:09,232 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:09,660 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:10,651 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:11,233 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:11,652 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:12,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:12,653 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:13,234 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:13,652 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:14,657 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:15,235 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:15,657 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:16,273 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:34:16,603 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:34:16,604 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:34:16,664 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:17,659 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:17,704 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:17,947 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:34:17,947 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:34:17,948 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:34:18,236 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:34:18,236 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:34:18,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:19,236 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:19,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:20,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:21,237 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:21,673 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:22,667 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:22,893 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:23,237 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:23,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:24,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:25,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:26,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:27,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:28,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:28,865 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:29,240 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:29,668 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:30,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:31,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:34:31,597 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:34:31,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:32,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:33,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:34,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:34,748 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:35,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:36,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:37,669 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:38,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:39,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:40,613 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:40,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:41,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:42,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:43,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:44,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:45,670 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:46,274 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:34:46,274 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:46,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:34:46,597 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:34:46,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:47,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:48,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:49,246 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:49,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:50,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:51,612 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:51,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:52,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:53,249 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:53,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:54,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:55,671 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:56,613 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:34:56,684 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:57,250 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:57,579 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:34:57,579 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 with policy now |
| 2025-01-21 21:34:57,584 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:34:57,659 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:34:57,659 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:34:57,673 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:57,686 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:34:57,808 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/yj9tkrjw-media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 |
| 2025-01-21 21:34:58,251 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:34:58,251 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 |
| 2025-01-21 21:34:58,251 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval |
| 2025-01-21 21:34:58,544 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/drleyppi-media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 |
| 2025-01-21 21:34:58,675 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:34:59,251 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:34:59,680 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:00,683 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:01,602 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:35:01,603 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:35:01,666 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:01,680 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:02,677 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:03,253 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:03,684 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:04,679 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:05,254 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:05,682 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:06,678 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:07,633 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:07,684 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:08,686 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:09,693 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:10,021 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:35:10,021 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:10,022 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:10,238 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 21:35:10,238 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval |
| 2025-01-21 21:35:10,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table |
| 2025-01-21 21:35:10,398 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:35:10,442 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/6i5c7b10-media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json |
| 2025-01-21 21:35:10,546 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/o6w4rf8n-events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:35:10,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:10,748 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-21 21:35:10,782 INFO wandb-upload_1:2651916 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpaetffev_ |
| 2025-01-21 21:35:10,793 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-21 21:35:11,207 INFO SenderThread:2651916 [sender.py:send_request_log_artifact():1525] logged artifact run-pkbouru1-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY3OTA1MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5NzUyOQ==', 'latestArtifact': None}} |
| 2025-01-21 21:35:11,207 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:35:11,207 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json with policy now |
| 2025-01-21 21:35:11,207 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:35:11,208 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,208 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,208 DEBUG SenderThread:2651916 [sender.py:send():379] send: telemetry |
| 2025-01-21 21:35:11,208 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,212 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,212 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,213 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,213 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,213 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,213 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,214 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,214 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,214 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,214 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,215 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,215 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,215 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,215 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,216 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,216 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,216 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,216 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,217 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,217 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:11,217 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:11,217 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-21 21:35:11,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:11,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:11,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:35:11,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json |
| 2025-01-21 21:35:11,257 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval |
| 2025-01-21 21:35:11,460 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/0rml6o24-media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json |
| 2025-01-21 21:35:11,483 INFO wandb-upload_1:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/ce38462j-media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json |
| 2025-01-21 21:35:11,604 INFO wandb-upload_2:2651916 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpuuksyuk_ |
| 2025-01-21 21:35:11,680 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:11,999 INFO SenderThread:2651916 [sender.py:send_request_log_artifact():1525] logged artifact run-pkbouru1-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1NzY3OTA2NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0NTE5NzUzMA==', 'latestArtifact': None}} |
| 2025-01-21 21:35:11,999 DEBUG SenderThread:2651916 [sender.py:send():379] send: files |
| 2025-01-21 21:35:11,999 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json with policy now |
| 2025-01-21 21:35:11,999 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:35:12,000 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:12,000 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:12,249 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/hmyxkqwf-media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json |
| 2025-01-21 21:35:12,256 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:12,680 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:13,001 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:13,257 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:13,680 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:14,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:15,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:16,274 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:35:16,597 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-21 21:35:16,597 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-21 21:35:16,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:17,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:18,665 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:18,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:19,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:20,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:21,681 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:22,682 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:23,682 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:24,616 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:24,682 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:25,409 DEBUG SenderThread:2651916 [sender.py:send():379] send: exit |
| 2025-01-21 21:35:25,409 INFO SenderThread:2651916 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-21 21:35:25,410 INFO SenderThread:2651916 [sender.py:send_exit():588] handling runtime: 279 |
| 2025-01-21 21:35:25,411 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:25,411 INFO SenderThread:2651916 [sender.py:send_exit():594] send defer |
| 2025-01-21 21:35:25,411 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:25,411 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-21 21:35:25,411 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:25,411 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-21 21:35:25,411 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-21 21:35:25,411 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:25,411 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-21 21:35:25,412 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:25,412 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-21 21:35:25,412 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-21 21:35:25,412 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:25,412 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-21 21:35:25,412 INFO HandlerThread:2651916 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-21 21:35:25,412 DEBUG SystemMonitor:2651916 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-21 21:35:25,412 DEBUG SystemMonitor:2651916 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-21 21:35:25,413 INFO HandlerThread:2651916 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-21 21:35:25,413 INFO HandlerThread:2651916 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-21 21:35:25,485 INFO HandlerThread:2651916 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-21 21:35:25,485 INFO HandlerThread:2651916 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-21 21:35:25,486 INFO HandlerThread:2651916 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-21 21:35:25,486 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:25,486 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-21 21:35:25,486 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-21 21:35:25,486 DEBUG SenderThread:2651916 [sender.py:send():379] send: stats |
| 2025-01-21 21:35:25,486 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:25,486 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-21 21:35:25,487 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:25,487 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-21 21:35:25,487 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-21 21:35:25,487 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:25,487 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-21 21:35:26,262 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:27,263 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:32,029 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:32,029 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:32,029 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-21 21:35:32,030 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:32,030 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-21 21:35:32,030 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-21 21:35:32,031 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:32,031 DEBUG SenderThread:2651916 [sender.py:send():379] send: history |
| 2025-01-21 21:35:32,031 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:32,031 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-21 21:35:32,031 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-21 21:35:32,032 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:32,032 DEBUG SenderThread:2651916 [sender.py:send():379] send: summary |
| 2025-01-21 21:35:32,036 INFO SenderThread:2651916 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-21 21:35:32,036 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:32,036 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-21 21:35:32,036 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-21 21:35:32,036 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:32,036 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-21 21:35:32,037 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:32,037 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-21 21:35:32,040 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-21 21:35:32,149 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-21 21:35:32,149 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:32,149 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-21 21:35:32,150 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:32,150 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-21 21:35:32,274 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/config.yaml |
| 2025-01-21 21:35:32,274 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:32,411 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:33,056 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-21 21:35:33,056 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:33,056 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:33,056 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-21 21:35:33,057 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:33,057 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-21 21:35:33,057 INFO SenderThread:2651916 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-21 21:35:33,059 INFO SenderThread:2651916 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-21 21:35:33,061 INFO SenderThread:2651916 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-21 21:35:33,076 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-21 21:35:33,077 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:33,077 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-21 21:35:33,077 DEBUG SenderThread:2651916 [sender.py:send():379] send: artifact |
| 2025-01-21 21:35:33,274 INFO Thread-12 :2651916 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:33,411 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:33,535 INFO wandb-upload_0:2651916 [upload_job.py:push():85] Skipped uploading /tmp/tmpjb9r83j6/wandb-job.json |
| 2025-01-21 21:35:33,535 INFO wandb-upload_1:2651916 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmphi4_s9v7 |
| 2025-01-21 21:35:33,630 INFO wandb-upload_2:2651916 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppc454fha |
| 2025-01-21 21:35:34,442 INFO SenderThread:2651916 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1NzY3OTQ4Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1NzY3MzQyMw==', 'versionIndex': 455}}} |
| 2025-01-21 21:35:34,442 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:34,442 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-21 21:35:34,442 INFO SenderThread:2651916 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-21 21:35:35,275 INFO SenderThread:2651916 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files |
| 2025-01-21 21:35:35,275 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/config.yaml config.yaml |
| 2025-01-21 21:35:35,275 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json wandb-summary.json |
| 2025-01-21 21:35:35,275 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/requirements.txt requirements.txt |
| 2025-01-21 21:35:35,279 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-21 21:35:35,279 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:35:35,279 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log output.log |
| 2025-01-21 21:35:35,281 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/diff.patch diff.patch |
| 2025-01-21 21:35:35,281 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-21 21:35:35,283 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 media/videos/eval/2_eval_2_6e61939f3ab9157666d6.mp4 |
| 2025-01-21 21:35:35,283 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 media/videos/eval/3_eval_3_cf7dc702773eb47d1f7e.mp4 |
| 2025-01-21 21:35:35,283 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 media/videos/eval/1_eval_1_335a125e2c942caab269.mp4 |
| 2025-01-21 21:35:35,284 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 media/videos/eval/4_eval_4_4fab844ed0fc1ded78b7.mp4 |
| 2025-01-21 21:35:35,284 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 media/videos/eval/0_eval_0_666330ccf994f224efc7.mp4 |
| 2025-01-21 21:35:35,284 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json media/table/eval/success_rate_ci_5_6f7b04d7609ef203f175.table.json |
| 2025-01-21 21:35:35,285 INFO SenderThread:2651916 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json media/table/eval/total_success_rate_ci_6_24085c194cbcf9ae6df8.table.json |
| 2025-01-21 21:35:35,285 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-21 21:35:35,285 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:35,285 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:35,285 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-21 21:35:35,285 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:35,285 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-21 21:35:35,285 INFO SenderThread:2651916 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:35:35,412 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:35,412 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:35,457 INFO wandb-upload_1:2651916 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/wandb-summary.json |
| 2025-01-21 21:35:35,480 INFO wandb-upload_0:2651916 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/config.yaml |
| 2025-01-21 21:35:35,502 INFO wandb-upload_2:2651916 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/requirements.txt |
| 2025-01-21 21:35:35,505 INFO wandb-upload_4:2651916 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/files/output.log |
| 2025-01-21 21:35:35,601 INFO wandb-upload_3:2651916 [upload_job.py:push():130] Uploaded file /tmp/tmp7bxqq5mnwandb/e2j3ddfv-events.out.tfevents.1737513046.leonmkim-ROG-Strix-G15CS-G15CS.2651732.0 |
| 2025-01-21 21:35:35,801 INFO Thread-11 (_thread_body):2651916 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-21 21:35:35,801 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:35,801 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-21 21:35:35,802 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:35,802 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-21 21:35:35,802 INFO SenderThread:2651916 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-21 21:35:35,803 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-21 21:35:35,803 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:35,803 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-21 21:35:35,803 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:35,803 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-21 21:35:35,803 INFO SenderThread:2651916 [file_stream.py:finish():601] file stream finish called |
| 2025-01-21 21:35:35,872 INFO SenderThread:2651916 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-21 21:35:35,872 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-21 21:35:35,872 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:35,872 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-21 21:35:35,873 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:35,873 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-21 21:35:35,873 INFO SenderThread:2651916 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-21 21:35:35,873 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-21 21:35:35,873 INFO HandlerThread:2651916 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-21 21:35:35,873 DEBUG SenderThread:2651916 [sender.py:send():379] send: final |
| 2025-01-21 21:35:35,873 DEBUG SenderThread:2651916 [sender.py:send():379] send: footer |
| 2025-01-21 21:35:35,873 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: defer |
| 2025-01-21 21:35:35,873 INFO SenderThread:2651916 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-21 21:35:35,874 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:35,874 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:35,875 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-21 21:35:35,875 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-21 21:35:35,875 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: server_info |
| 2025-01-21 21:35:35,875 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-21 21:35:35,877 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-21 21:35:35,877 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-21 21:35:35,925 DEBUG SenderThread:2651916 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-21 21:35:35,925 INFO MainThread:2651916 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-21 21:35:35,925 INFO MainThread:2651916 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-21 21:35:35,926 INFO MainThread:2651916 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-21 21:35:35,926 DEBUG HandlerThread:2651916 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-21 21:35:35,926 INFO HandlerThread:2651916 [handler.py:finish():882] shutting down handler |
| 2025-01-21 21:35:36,875 INFO WriterThread:2651916 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/4295_2/pkbouru1/wandb/run-20250121_213045-pkbouru1/run-pkbouru1.wandb |
| 2025-01-21 21:35:36,925 INFO SenderThread:2651916 [sender.py:finish():1615] shutting down sender |
| 2025-01-21 21:35:36,925 INFO SenderThread:2651916 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-21 21:35:36,925 INFO SenderThread:2651916 [file_pusher.py:join():175] waiting for file pusher |
|
|
